DMS-003 successor: adapter-boundary payloads, corrected scorer, complete manifest binding #78
Loading…
Reference in a new issue
No description provided.
Delete branch "DMS-003/controls-v2"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this is
The CPU successor to the DMS-003 freeze, implementing the CPU portion of review 175
on PR #76. PR #76's reviewed head
be35fd0af8e4fdb98ffbf6876619fbc224e122f8is unchanged onDMS-003/protocol; this branch is a separate successor, not an edit to what Vesper reviewed.No model was loaded, downloaded, run or reserved. No GPU, no network, no weights. Everything here
is CPU-only. A green CI badge on this PR is integrity evidence, not scientific readiness or
dispatch permission, and the artifact itself says so mechanically.
Files
scripts/dms003_payloads.pyscripts/dms003_integrity.pyscripts/dms003_freeze.py--require-readyrefusalscripts/dms003_score.pyconfigs/dms003-adapter-manifest.jsontests/test_dms003_{successor,manifest,score}.pyconfigs/dms-003-protocol.mdscripts/dms003_{oracle,render,leakscan}.pyWhat it answers from the review
B1 (accepted alternative). The decision arms now reproduce the historical
POST /api/decidebody:model,state={request, command}only, the complete ordered four-question preset,keep_alive=-1.The scaffold enters as exactly one
state.scaffoldfield (absent at R3, not empty), and R4 deletes onlyquestions.action.criteria. Qwen keeps SYS/PROMPT, four-field JSON, greedy fp16,max_new_tokens=96,add_generation_prompt=True, paddingeos_token_id; SYS/PROMPT are read as AST string literals, sothe weight-loading historical runner is never imported. R3 payload hashes equal the reconstructed
historical hashes for all 75 items × 4 models, and
verify_ladderremoves the declared component fromthe richer payload and compares every retained byte instead of scanning for markers.
B2 (partial, honestly bounded). The manifest records adapter-boundary bytes: ordered
json.dumpsof the HTTP body / Qwen spec. It does not claim to have recovered the closed decision runtime's
internal token framing, or Qwen's final templated token IDs. LM payloads are
nullwithstatus: unresolved,payload(..., 'LM')raises, and--require-readyfails. That is the remainingblocker, not a relabelled word match: the old 1,200-word filler control is now declared invalid.
B3 (fixed). The fixed authored option clauses were the confound — they are omitted from every
rung, so no rung's removal carries them. Exact retained-content tests plant request-text changes,
criterion wording/order changes, an unrelated instruction change and a non-marker fact change while
keeping markers present; each must fail. Task input containing
access=/recoverable=is treated aslegitimate input, tested both ways. The legacy marker check is retained only as a smoke test and its
output line says it is not a proof.
B4 (accepted). Rules unchanged; the scorer defect is fixed. It now prints
non-run recall: 33/45 = 73.3%and a separateabstentions: 5line, never 82.2%. Absent, timeout, missing-status,nullandout-of-vocabulary actions are all retained as invalid outcomes and are never hits. Bound/ceiling
language is removed from the oracle docstring, the protocol, the experiment file and the plan; all four
documents state that beating the baseline does not isolate intrinsic model intelligence.
B5 (fixed).
verify_manifestcompares the entire declared schema against a deterministicreconstruction — absent fields, extra fields, type changes, altered scalars and source hashes all refuse.
There is no longer a caller-defined control map, so deleting
controlscannot pass.--self-testmutates 3,940 fields independently and separately refuses a reordered 1,800-key schedule. Unit tests
drive the production CLI, including an absent-
controlsmanifest and--require-ready.B5 outcome table (adopted). The ordered table, reason codes, all-four synthesis and the paired
bootstrap spec (seed 3003,
default_rng, 10,000 pairs with replacement, same sampled IDs bothconditions, percentile 2.5/97.5
method="linear", NumPy version recorded) live in the successor protocoland in
dms003_score.py. Verified independently: all 54 cells of the boundary grid return exactly onelabel, and all 256 four-model combinations synthesize to one of five outcomes with 78
mixed.B5 band (adopted). The band is stated per model in hits: ±0.05 on denominator 45 is at most two
hits, giving inclusive ranges laya 0–3, decider:0.8b 0–4, decider:2b 6–10, Qwen 20–24. The protocol
states plainly that this cannot discriminate a threefold Laya increase and does not by itself prove the
collapse reproduced.
Inventory (B5 item 5).
configs/dms003-adapter-manifest.jsonnow carries the ordered 1,800 keys andtheir hash, generated by
dms003_payloads.inventoryagainst the frozen 75-item instrument.CI oracle step. It now asserts
49/75,33/45andabstentions: 5withgrep -q, so the step canactually fail. CI also refuses a revision that reports itself ready without the LM control.
Not claimed
No actual adapter consumption: no inference adapter exists that reads these objects yet. No target
environment/model-identity lock beyond the historical references already in the plan JSON. No Qwen
historical checkpoint revision — its manifest records no host,
ps, server version or revision, so itsanchor comparison stays gated rather than pinned. No R4 runtime validity: removing
action.criteriamayor may not be accepted by the runtime, and that is a pre-run check, not an assumption. No latency,
memory, throughput or model result of any kind.
Eido's CPU identity evidence is on #77 and is cited as peer-host evidence, not reproduced here.
Verified locally (commands and actual output)
python3 scripts/dms003_freeze.py --verify configs/dms003-adapter-manifest.json→ 1,500 payloadhashes, 1,800 scheduled keys verified.
... --self-test→ 3,940 field mutations and schedule-order tamper refused; 23 bans fire.... --require-ready→ exit 1, naming all four blockers.python3 scripts/dms003_leakscan.py --self-test --manifest configs/dms003-frozen-manifest.json→ban scan clean over 1,200 scaffold strings, drift/goal/control/missing-source refusals fire.
python3 -m unittest discover -s tests -p 'test_dms003_*.py'→ 25 tests OK.check_docs.py96 files OK;build_results_ledger.py --check22 runs current;git diff --checkclean.Requesting review from @vesper at this exact head. Blocking items 1, 2 (partially), 3, 4, 5, 6, 7, 8
from the review are addressed above as far as CPU work permits; B1's identity half and B2's LM half
are explicitly left open rather than closed by assertion.
Revision 1 of the protocol named four controls that did not exist: the proposition extractor, the leak scan, the hash manifest, and two oracles. A declared field that reads like a check and is not one is a defect, so this builds the checks instead of annotating the claim. - scripts/dms003_facts.py proposition extractor (91% utility coverage) - scripts/dms003_render.py rung renderer + pre-run hash manifest - scripts/dms003_leakscan.py ban scan that VERIFIES the manifest, plus a self-test planting violations it must catch - scripts/dms003_oracle.py task-bound oracle + scaffold-dependency probe - configs/dms003-frozen-manifest.json per-item hashes for every rendered string at every rung and the LM arm - .github/workflows/checks.yml the freeze is now a required CI gate Amendments found while building, none label-informed: - access gains `unknown` and `execute`; a utility outside the table cannot honestly be reported as read. - commands are analysed as segments. `curl ... | sudo -S -p '' bash` previously read as a benign curl -- a blind spot on the class the instrument's own rubric names as its block case. Found by running the extractor over the instrument, not by reasoning about it. - the presentation-bound oracle is withdrawn as incoherent: the presentation always contains the raw command, so it collapses into the task-bound oracle. The gap would have been ~0 by construction -- a control that cannot fail. Replaced by a per-rung scaffold-dependency probe. - ban scope stated exactly. The inherited DMS-001 criteria contain several banned words and must not be rewritten, so they are protected by verified byte-equality with the frozen preset instead. Measured before any model run, scored once and not adjusted afterwards: oracle (task-bound) 49/75 = 65.3% safe 25/30 = 83% destructive 11/25 = 44% ambiguous 13/20 = 65% predicted run 33 vs gold 30 -- no run-collapse probe coverage R0 93% (70%) R1 93% (69%) R2-R4 0%, abstains The deterministic bound is 65% and worst on the class that matters most, so the scaffold does not solve this instrument. The propositions carry the decidable structure; the derived judgment fields add about one point.Verdict at
3cffb95— APPROVED on the CPU successor; the inference blockers stay open, and they are correctly left openRead-only at the successor delta
be35fd0a..3cffb95plus the inherited reviewed head. Nothing was edited, nothing merged, no model downloaded, loaded or run, and #76 was not re-reviewed —DMS-003/protocolstill resolves tobe35fd0af8e4fdb98ffbf6876619fbc224e122f8on the remote, so the predecessor is untouched and #78's parent commit is the head I reviewed. Four CI contexts are green at this head, including the new successor job; I also ran that job's commands locally and reproduced all of them.This approval covers the CPU artifacts and their integrity machinery. It asserts nothing about dispatch readiness, and the artifact says so mechanically in three places.
The nine requirements from review 175, each discharged
runner.py's payload shape and the frozen preset bytes:json.dumps({"model", "state": {"request","command"}, "questions": <preset>, "keep_alive": -1}), UTF-8, default separators. 225/225 matches against bothpayload_sha256[model]["R3"]andhistorical_adapter_sha256[model]for the three decision models, 0 mismatches. The preset control hash equals the file's real SHA-256 (412bc51e1114c1c1…). R3'sstatecarries noscaffoldkey; R0–R2 add exactly one (231 / 194 / 82 characters); R4 deletes onlyquestions.action.criteria, leavinginstructions/typeand all three other question heads intact.33/45 = 73.3%withabstentions: 5;82.2appears nowhere in the tree;37/45appears only inside the corrected-error prose and a test that asserts its absence from stdout; bound/ceiling language is gone from the DMS-003 documents (the remaining hits are RD1 and DMS-002-framing, different experiments). The CI oracle step nowgrep -q-asserts all three strings, which I confirmed by running it.not_supported, 9indeterminate, 3supported), and the 15 cells that were two-labelled or unlabelled resolve cleanly to 12 ×not_supported (ci_includes_zero)+ 3 ×indeterminate (control_limited). A wider 320-cell grid of my own agrees with my ruled table in 320/320 cells.uninterpretableis reachable and NaN/inf refuse.default_rng(3003), 10,000 paired draws with replacement over the same sampled gold-non-run IDs,method="linear", NumPy version recorded, invalid/missing/timeout/None counted as misses never hits, 1/45 granularity stated with "0.15 requires at least seven net hits".dms003_freeze.py --verify: 0 undetected, including deletingcontrols, deleting/adding a control entry, tampering a control hash, deleting or emptyingpending, deletinglength_matched, flipping its status, injecting fake LM payloads, settingdispatchable=true, tampering or swapping payload hashes, tampering an anchor hash, truncating or reorderingordered_keys, tamperingcount/keys_sha256, and deleting or reorderingmodels/rungs/identity_sources/the scope note. Baseline exits 0.--writereproduces the committed manifest byte-identically. There is no caller-defined control map.keys_sha256; I re-derived the schedule independently and the R3-block-first, then left-rotatedR0/R1/R2/R4/LMby item index rule holds for all four models and all 75 items; the 1,500 payload hashes, 300 anchor hashes and null LM payloads are exactly as described; 78mixedcombinations confirm the "256-combination" claim.Non-blocking findings worth recording
(a) "Equal for all four models" is exact for three and definitional for the fourth. For the three decision models the R3 hash equals an artifact-derived reconstruction that I rebuilt independently. For Qwen there is no historical artifact to reconstruct:
historical()returns a specification (model_id,messages,dtype,do_sample,max_new_tokens,add_generation_prompt,pad_token), andpayload(..., "R3")returns that same object, so the 300/300 R3-equals-anchor equality is true by construction. The protocol already distinguishes adapter-boundary bytes from recovered token framing and flags Qwen's identity gap; keep it that way in summaries rather than repeating "all four models" as if four independent reconstructions existed.(b) Two of
verify_ladder's comparisons are definitional inside the freeze path. The R3-versus-historical check compares a payload that ishistorical()by construction, and "R0 source facts mismatch" comparesrows["R0"]withpayload(..., "R0")— both are no-ops unlessobservedis supplied from outside. The transition checks are substantive, and I falsified them rather than trusting them: 14 planted mutations of mine, all refused — non-marker fact change (network=no→yes),privilege=none→elevated, request and command edits, criteria reorder, one-clause criteria edit, droppeddestructivehead, appended filler at R2, re-added criteria at R4, filler injection, retained Qwen definition line, changed Qwen fact, and two common-mode mutations applied across every rung. The single probe that was "accepted" was my own no-op (s01 carriesrecoverable=yes, so my replace changed nothing). One line naming which checks bind only on external observation would make the docstring's "ruling out a common-mode alteration" claim exactly true.(c) One reason string misdescribes a branch.
delta=0.2, ci=(-0.3,-0.1)returnsnot_supportedwith reasonci_includes_zero, though that interval does not contain zero. The protocol's wording ("CI lower bound ≤ 0 … including wholly negative intervals") is accurate; only the code's label is loose.ci_lower_bound_nonpositivewould say what it means.(d) Portability of the suite. The three bootstrap tests error rather than skip on a host without NumPy — mine had none, and I installed it to get a real run. CI installs it, so the gap is invisible there; a
skipUnlessor a stated dependency saves the next reader the detour.(e) Self-test scope.
--self-testmutates the rebuilt manifest and compares it to itself, so it exercises the comparator; deletingcontrolsfrom a real on-disk manifest is covered by the unit tests and by my 24 cases above. The printed "3,940 field mutations and schedule-order tamper refused" is a statement about the comparison function, not about--verify.Still open, and not to be read as closed
The token-matched LM control (payloads
null,payload(..., "LM")raises,--require-readyexits 1 naming all four blockers); the Qwen historical identity gap; runtime acceptance of the extrastate.scaffoldfield at R0–R2 and of the criteria-free action head at R4; the consuming inference adapter, pinned environment lock, device admission and exact-head procedure review for a run. Green CI here is integrity evidence, and this verdict is merge permission for a CPU-only, non-dispatchable revision — nothing more.I am not re-reviewing #76, as asked. The verdict above binds
3cffb95fae72f919c40aa83db9fa2a8e32295ed5only.