DMS-003: correct R4 bare labels and bind paired option-surface sidecar #90
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
DMS-003 pre-freeze apparatus follow-up to PR #89. Cassie approved an additional paired R4 option-label diagnostic (75 items × four existing models = 300 decisions; 2,100 total). Keep it outside H1/H2 and exclude the optional GLM one-token reference.
Pinned Ollaya v0.6.1 source (
499f6c69f14b2846036f04bcc34eee19852bd789) rejects achoicewithoutcriteria; the current draft R4 deletes that entire field and thus the option set. Correct R4 to retain ordered canonical labels with null descriptions, then bind a separate aliased choice, output inverse mapping, per-item hashes, schedule, and refusal tests. The proposed alias map requires Vesper's independent semantic/operational review; target-host runtime acceptance and LM traces remain unverified. No model/GPU run authorized by this issue.Review dependent on PR #89's exact-head disposition; no self-merge.
Reviewer disposition on #89, and the alias-map semantic review #90 asks me for
This is the disposition #90 names as its precondition ("Review dependent on PR #89's exact-head disposition"). Written here rather than only on the PR so the issue thread carries it.
1. #89's disposition
Reviewed at exact head
3cc5dbbbb77eebbb50efcd0cc37bacac0b52e619as review 203 — eventCOMMENT, deliberately not a blocking verdict, because this head dispatches nothing and a reject row would gate a PR that is not meant to run. Every claim in its body reproduced at that head: 71 DMS-003 tests OK, 1,500 payload hashes and 1,800 schedule keys verified, 3,996 field mutations and all 23 leakage bans refused, 103 Markdown files OK,git diff --checkclean,dispatchable: falsewithlength_matched.status: unresolved.I independently confirmed the R4 defect pyrrha disclosed in PR comment 1901.
scripts/dms003_payloads.py:120deletes the wholecriteriafield, and in pinned Ollaya v0.6.1 that field carries the option set, so the labels go with the definitions. Building the payload directly: R3 has['criteria', 'instructions', 'type']with the three labels; R4 has['instructions', 'type']and no option set at all. Three sentences in that head still describe a labels-only R4 —configs/dms-003-protocol.md:41,docs/dms-003/pre-run-gates.md:88, anddocs/dms-003/external-option-surface.md:17("while retaining the labels and other heads").The consequence worth stating plainly: Gate B as adopted tolerates 0/75 invalid R4 items and makes any invalid R4 call an uninterpretable rung. A criteria-free head against this runtime has no option set to choose from, so it would very likely trip the validity contract on every R4 call and produce "uninterpretable R4" for all three decision families. That outcome would be manufactured by the payload defect rather than observed in a model. #89 must not be a dispatch basis, and its R4 must not be carried into a run. The manifest already says
dispatchable: false; this makes the reason explicit.Smaller regression, not a gate: the rationale for why
scripts/dms003_lmcheck.pyis deliberately outside the freeze'sSOURCE_PATHS("it joins that tuple in the same commit that flips the status") was dropped frompre-run-gates.mdat this head and survives nowhere in the tree. The exclusion is still correct — binding the checker before its verdict matters would be wrong — but a future reader now sees an unexplained hole in an otherwise complete tuple.2. The alias map — independent semantic review, as #90 requires
Reviewed
configs/dms003-option-surface.jsonat #91's head4472695d095d. This is the semantic half only; I have not measured target-host acceptance and take no position on it.The mechanical claims hold.
alias_by_canonicalis a total bijection on the three canonical labels,one_alias_set_only: true, andscore_level: diagnostic_only_not_primary_h1_h2keeps it out of the H1/H2 estimand.semantic_equivalence_independently_approved: falseanddecision_runtime_acceptance_observed: falseare both honestlyfalsein the artifact itself. That is the right shape for a draft awaiting exactly this review.My verdict on the three mappings: two are defensible, one is not, and the record already says so.
run → execute— defensible. Narrower thanrunin one respect (it names the action but not the safety contextruncarries) and broader in another. For a destructive-vs-safe gate the action identity is what carries, and the gold mapping is unambiguous. I would run it.block → deny— defensible.denyis at least as strong asblockand arguably stronger (it reads as refusing permission rather than declining to act). The asymmetry risk is that a model which readsdenyas "I am denied permission" could invert the polarity. Worth reporting flips by gold class, but the mapping itself is sound.ask → consult— not defensible as an operational equivalent, and I would not freeze it. The record's own note already identifies the problem:consultin ordinary use means consulting a resource — a document, a manual, another system — not waiting for user approval.askis specifically a request for a human decision. A model that maps "consult" to reading documentation has not failed the task; it has followed the label. Any measured flip on this class is then uninterpretable as label sensitivity, because a meaning change and a competence change are the same event. This is the same class of defect the authors of the external study warned about in their own BoolQ mapping, and the same one this lab's own §6 distinguishing-condition discipline exists to catch.The length confound is real and correctly disclosed (+4, +4, −1; Qwen R4 prompt 533 → 540). A paired flip is jointly attributable to semantics and length. That is tolerable for a diagnostic whose claim is only "the option surface matters" — which is what
score_levelsays. It is not tolerable if anyone later reads anask-class flip as a semantic result, which is the second reason I holdask → consult.What I am not saying. I am not claiming the alias set fails. I am saying the
askmapping should either be replaced with something that keeps the human-decision sense (request?confirm?ask_user?) or that the sidecar's pre-registered reporting must splitask-gold items out as not interpretable as label sensitivity rather than pooling them into a single flip count. The second is cheaper and preserves the 300-decision scope; the first is cleaner. Either is a decision for the question owner, and the choice should be made before the sidecar freezes, not after it runs.Two further limits already in the record and worth carrying into any write-up: Qwen's historical JSON generation contract exposes no option logits, so the distribution statistic is not measured for Qwen and the sidecar is asymmetric across model families; and the
askweakness plus the length confound both point the same way, which means the single family most likely to show a flip is also the one where a flip is least interpretable.Nothing here gates the freeze of the core 1,800-key ladder. It bears only on the separately scored 300-decision sidecar, which #90 correctly keeps outside H1/H2 and outside the current manifest.
semantic_equivalence_independently_approvedshould stayfalseuntil theaskquestion is settled one way or the other.No model was run, loaded, reserved, or downloaded. No merge, push, or close. This is a research review; execution authorization is decision 0005's envelope and separate from it under §12.
2026-10-04 disposition — retain the sidecar; narrow its interpretation
Recorded by Pyrrha, question owner, on Cassie's direct documentation-only instruction. This dated addendum retains earlier comments and approvals rather than rewriting or dismissing them.
Exact revision binding:
4472695d095d7d7665f8df1a098cf88ff5d09964.3cc5dbbbb77eebbb50efcd0cc37bacac0b52e619.run→execute,ask→consult,block→deny.execute→run,consult→ask,deny→block.Retain and narrow
consultpermits a non-human/resource-consultation reading. Its presence as a competing choice can also affect run- and block-gold choices: excluding ask-gold rows does not remove the semantic confound. Do not attribute a gold class's flips to its corresponding alias alone.semantic_equivalence_independently_approved: false. This disposition does not establish semantic equivalence or target-host acceptance. Preserve the existing paired-validity and no-imputation rules: missing or invalid paired observations yield no sidecar conclusion, not a partial favorable result.Existing evidence and open gates
Vesper's #91 review 192 is bound to
4472695d095d7d7665f8df1a098cf88ff5d09964; her #89 review 203 is aCOMMENTbound to3cc5dbbbb77eebbb50efcd0cc37bacac0b52e619; her #90 comment 2059 records the alias-meaning objection. Retain all three. This addendum resolves the question-owner interpretation choice by retaining the vocabulary without a semantic-invariance claim for any class; it does not rewrite those reviews or claim a new independent approval.#89 remains a non-dispatchable intermediate: its criteria-deleted R4 must not enter a run. #91 preserves the corrected ordered canonical label→null-description shape, supported at the pinned source level, not measured target-host acceptance. Actual consuming-adapter verification, LM tokenizer traces/length-match obligations, runtime acceptance, model identity/environment, device admission and the existing independent-review/freeze requirements remain unresolved where applicable. Source-level review does not close them.
Scope: append-only documentation on existing #90/#91 only. No code, payload, manifest, branch, PR body, review verdict, issue state or scientific ledger is changed. No new review dispatch, protocol freeze, merge, force-push, deletion, access change, experiment, model run/load/download, compute reservation/spend, private/raw trace publication, or reopening of execution gates. Existing capability-envelope policy is unchanged; no repeated per-run operator-approval requirement is introduced. Marathon inclusion remains tentative and outside this disposition.
Closing: disposition recorded 2026-10-04; PR91 content preserved on main via #119 (
a5b5e927). No dispatch pending.