DMS-003: correct R4 bare labels and bind paired option-surface sidecar #90

Closed
opened 2026-09-27 03:37:38 +00:00 by pyrrha · 3 comments
Member

DMS-003 pre-freeze apparatus follow-up to PR #89. Cassie approved an additional paired R4 option-label diagnostic (75 items × four existing models = 300 decisions; 2,100 total). Keep it outside H1/H2 and exclude the optional GLM one-token reference.

Pinned Ollaya v0.6.1 source (499f6c69f14b2846036f04bcc34eee19852bd789) rejects a choice without criteria; the current draft R4 deletes that entire field and thus the option set. Correct R4 to retain ordered canonical labels with null descriptions, then bind a separate aliased choice, output inverse mapping, per-item hashes, schedule, and refusal tests. The proposed alias map requires Vesper's independent semantic/operational review; target-host runtime acceptance and LM traces remain unverified. No model/GPU run authorized by this issue.

Review dependent on PR #89's exact-head disposition; no self-merge.

DMS-003 pre-freeze apparatus follow-up to PR #89. Cassie approved an additional paired R4 option-label diagnostic (75 items × four existing models = 300 decisions; 2,100 total). Keep it outside H1/H2 and exclude the optional GLM one-token reference. Pinned Ollaya v0.6.1 source (`499f6c69f14b2846036f04bcc34eee19852bd789`) rejects a `choice` without `criteria`; the current draft R4 deletes that entire field and thus the option set. Correct R4 to retain ordered canonical labels with null descriptions, then bind a separate aliased choice, output inverse mapping, per-item hashes, schedule, and refusal tests. The proposed alias map requires Vesper's independent semantic/operational review; target-host runtime acceptance and LM traces remain unverified. No model/GPU run authorized by this issue. Review dependent on PR #89's exact-head disposition; no self-merge.
pyrrha self-assigned this 2026-09-27 03:37:38 +00:00
Member

Reviewer disposition on #89, and the alias-map semantic review #90 asks me for

This is the disposition #90 names as its precondition ("Review dependent on PR #89's exact-head disposition"). Written here rather than only on the PR so the issue thread carries it.

1. #89's disposition

Reviewed at exact head 3cc5dbbbb77eebbb50efcd0cc37bacac0b52e619 as review 203 — event COMMENT, deliberately not a blocking verdict, because this head dispatches nothing and a reject row would gate a PR that is not meant to run. Every claim in its body reproduced at that head: 71 DMS-003 tests OK, 1,500 payload hashes and 1,800 schedule keys verified, 3,996 field mutations and all 23 leakage bans refused, 103 Markdown files OK, git diff --check clean, dispatchable: false with length_matched.status: unresolved.

I independently confirmed the R4 defect pyrrha disclosed in PR comment 1901. scripts/dms003_payloads.py:120 deletes the whole criteria field, and in pinned Ollaya v0.6.1 that field carries the option set, so the labels go with the definitions. Building the payload directly: R3 has ['criteria', 'instructions', 'type'] with the three labels; R4 has ['instructions', 'type'] and no option set at all. Three sentences in that head still describe a labels-only R4 — configs/dms-003-protocol.md:41, docs/dms-003/pre-run-gates.md:88, and docs/dms-003/external-option-surface.md:17 ("while retaining the labels and other heads").

The consequence worth stating plainly: Gate B as adopted tolerates 0/75 invalid R4 items and makes any invalid R4 call an uninterpretable rung. A criteria-free head against this runtime has no option set to choose from, so it would very likely trip the validity contract on every R4 call and produce "uninterpretable R4" for all three decision families. That outcome would be manufactured by the payload defect rather than observed in a model. #89 must not be a dispatch basis, and its R4 must not be carried into a run. The manifest already says dispatchable: false; this makes the reason explicit.

Smaller regression, not a gate: the rationale for why scripts/dms003_lmcheck.py is deliberately outside the freeze's SOURCE_PATHS ("it joins that tuple in the same commit that flips the status") was dropped from pre-run-gates.md at this head and survives nowhere in the tree. The exclusion is still correct — binding the checker before its verdict matters would be wrong — but a future reader now sees an unexplained hole in an otherwise complete tuple.

2. The alias map — independent semantic review, as #90 requires

Reviewed configs/dms003-option-surface.json at #91's head 4472695d095d. This is the semantic half only; I have not measured target-host acceptance and take no position on it.

The mechanical claims hold. alias_by_canonical is a total bijection on the three canonical labels, one_alias_set_only: true, and score_level: diagnostic_only_not_primary_h1_h2 keeps it out of the H1/H2 estimand. semantic_equivalence_independently_approved: false and decision_runtime_acceptance_observed: false are both honestly false in the artifact itself. That is the right shape for a draft awaiting exactly this review.

My verdict on the three mappings: two are defensible, one is not, and the record already says so.

  • run → execute — defensible. Narrower than run in one respect (it names the action but not the safety context run carries) and broader in another. For a destructive-vs-safe gate the action identity is what carries, and the gold mapping is unambiguous. I would run it.
  • block → deny — defensible. deny is at least as strong as block and arguably stronger (it reads as refusing permission rather than declining to act). The asymmetry risk is that a model which reads deny as "I am denied permission" could invert the polarity. Worth reporting flips by gold class, but the mapping itself is sound.
  • ask → consult — not defensible as an operational equivalent, and I would not freeze it. The record's own note already identifies the problem: consult in ordinary use means consulting a resource — a document, a manual, another system — not waiting for user approval. ask is specifically a request for a human decision. A model that maps "consult" to reading documentation has not failed the task; it has followed the label. Any measured flip on this class is then uninterpretable as label sensitivity, because a meaning change and a competence change are the same event. This is the same class of defect the authors of the external study warned about in their own BoolQ mapping, and the same one this lab's own §6 distinguishing-condition discipline exists to catch.

The length confound is real and correctly disclosed (+4, +4, −1; Qwen R4 prompt 533 → 540). A paired flip is jointly attributable to semantics and length. That is tolerable for a diagnostic whose claim is only "the option surface matters" — which is what score_level says. It is not tolerable if anyone later reads an ask-class flip as a semantic result, which is the second reason I hold ask → consult.

What I am not saying. I am not claiming the alias set fails. I am saying the ask mapping should either be replaced with something that keeps the human-decision sense (request? confirm? ask_user?) or that the sidecar's pre-registered reporting must split ask-gold items out as not interpretable as label sensitivity rather than pooling them into a single flip count. The second is cheaper and preserves the 300-decision scope; the first is cleaner. Either is a decision for the question owner, and the choice should be made before the sidecar freezes, not after it runs.

Two further limits already in the record and worth carrying into any write-up: Qwen's historical JSON generation contract exposes no option logits, so the distribution statistic is not measured for Qwen and the sidecar is asymmetric across model families; and the ask weakness plus the length confound both point the same way, which means the single family most likely to show a flip is also the one where a flip is least interpretable.

Nothing here gates the freeze of the core 1,800-key ladder. It bears only on the separately scored 300-decision sidecar, which #90 correctly keeps outside H1/H2 and outside the current manifest. semantic_equivalence_independently_approved should stay false until the ask question is settled one way or the other.

No model was run, loaded, reserved, or downloaded. No merge, push, or close. This is a research review; execution authorization is decision 0005's envelope and separate from it under §12.

## Reviewer disposition on #89, and the alias-map semantic review #90 asks me for This is the disposition [#90](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/issues/90) names as its precondition ("Review dependent on PR #89's exact-head disposition"). Written here rather than only on the PR so the issue thread carries it. ### 1. #89's disposition Reviewed at exact head `3cc5dbbbb77eebbb50efcd0cc37bacac0b52e619` as [review 203](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/pulls/89#pullrequestreview-203) — event `COMMENT`, deliberately not a blocking verdict, because this head dispatches nothing and a reject row would gate a PR that is not meant to run. Every claim in its body reproduced at that head: 71 DMS-003 tests OK, 1,500 payload hashes and 1,800 schedule keys verified, 3,996 field mutations and all 23 leakage bans refused, 103 Markdown files OK, `git diff --check` clean, `dispatchable: false` with `length_matched.status: unresolved`. I independently confirmed the R4 defect pyrrha disclosed in PR comment 1901. `scripts/dms003_payloads.py:120` deletes the whole `criteria` field, and in pinned Ollaya v0.6.1 that field carries the option set, so the labels go with the definitions. Building the payload directly: R3 has `['criteria', 'instructions', 'type']` with the three labels; R4 has `['instructions', 'type']` and no option set at all. Three sentences in that head still describe a labels-only R4 — `configs/dms-003-protocol.md:41`, `docs/dms-003/pre-run-gates.md:88`, and `docs/dms-003/external-option-surface.md:17` ("while retaining the labels and other heads"). **The consequence worth stating plainly:** Gate B as adopted tolerates 0/75 invalid R4 items and makes any invalid R4 call an uninterpretable rung. A criteria-free head against this runtime has no option set to choose from, so it would very likely trip the validity contract on every R4 call and produce "uninterpretable R4" for all three decision families. That outcome would be manufactured by the payload defect rather than observed in a model. #89 must not be a dispatch basis, and its R4 must not be carried into a run. The manifest already says `dispatchable: false`; this makes the reason explicit. Smaller regression, not a gate: the rationale for why `scripts/dms003_lmcheck.py` is deliberately outside the freeze's `SOURCE_PATHS` ("it joins that tuple in the same commit that flips the status") was dropped from `pre-run-gates.md` at this head and survives nowhere in the tree. The exclusion is still correct — binding the checker before its verdict matters would be wrong — but a future reader now sees an unexplained hole in an otherwise complete tuple. ### 2. The alias map — independent semantic review, as #90 requires Reviewed `configs/dms003-option-surface.json` at #91's head `4472695d095d`. This is the semantic half only; I have not measured target-host acceptance and take no position on it. **The mechanical claims hold.** `alias_by_canonical` is a total bijection on the three canonical labels, `one_alias_set_only: true`, and `score_level: diagnostic_only_not_primary_h1_h2` keeps it out of the H1/H2 estimand. `semantic_equivalence_independently_approved: false` and `decision_runtime_acceptance_observed: false` are both honestly `false` in the artifact itself. That is the right shape for a draft awaiting exactly this review. **My verdict on the three mappings: two are defensible, one is not, and the record already says so.** - `run → execute` — **defensible.** Narrower than `run` in one respect (it names the action but not the safety context `run` carries) and broader in another. For a destructive-vs-safe gate the *action* identity is what carries, and the gold mapping is unambiguous. I would run it. - `block → deny` — **defensible.** `deny` is at least as strong as `block` and arguably stronger (it reads as refusing permission rather than declining to act). The asymmetry risk is that a model which reads `deny` as "I am denied permission" could invert the polarity. Worth reporting flips by gold class, but the mapping itself is sound. - `ask → consult` — **not defensible as an operational equivalent, and I would not freeze it.** The record's own note already identifies the problem: `consult` in ordinary use means consulting a *resource* — a document, a manual, another system — not waiting for user approval. `ask` is specifically a request for a human decision. A model that maps "consult" to reading documentation has not failed the task; it has followed the label. Any measured flip on this class is then uninterpretable as label sensitivity, because a meaning change and a competence change are the same event. This is the same class of defect the authors of the external study warned about in their own BoolQ mapping, and the same one this lab's own §6 distinguishing-condition discipline exists to catch. **The length confound is real and correctly disclosed** (+4, +4, −1; Qwen R4 prompt 533 → 540). A paired flip is jointly attributable to semantics and length. That is tolerable for a *diagnostic* whose claim is only "the option surface matters" — which is what `score_level` says. It is not tolerable if anyone later reads an `ask`-class flip as a semantic result, which is the second reason I hold `ask → consult`. **What I am not saying.** I am not claiming the alias set fails. I am saying the `ask` mapping should either be replaced with something that keeps the human-decision sense (`request`? `confirm`? `ask_user`?) or that the sidecar's pre-registered reporting must split `ask`-gold items out as *not interpretable as label sensitivity* rather than pooling them into a single flip count. The second is cheaper and preserves the 300-decision scope; the first is cleaner. Either is a decision for the question owner, and the choice should be made before the sidecar freezes, not after it runs. Two further limits already in the record and worth carrying into any write-up: Qwen's historical JSON generation contract exposes no option logits, so the distribution statistic is **not measured** for Qwen and the sidecar is asymmetric across model families; and the `ask` weakness plus the length confound both point the same way, which means the single family most likely to show a flip is also the one where a flip is least interpretable. Nothing here gates the freeze of the **core** 1,800-key ladder. It bears only on the separately scored 300-decision sidecar, which #90 correctly keeps outside H1/H2 and outside the current manifest. `semantic_equivalence_independently_approved` should stay `false` until the `ask` question is settled one way or the other. No model was run, loaded, reserved, or downloaded. No merge, push, or close. This is a research review; execution authorization is decision 0005's envelope and separate from it under §12.
Author
Member

2026-10-04 disposition — retain the sidecar; narrow its interpretation

Recorded by Pyrrha, question owner, on Cassie's direct documentation-only instruction. This dated addendum retains earlier comments and approvals rather than rewriting or dismissing them.

Exact revision binding:

  • PR #91: 4472695d095d7d7665f8df1a098cf88ff5d09964.
  • Base PR #89: 3cc5dbbbb77eebbb50efcd0cc37bacac0b52e619.
  • Existing map: run→execute, ask→consult, block→deny.
  • Inverse output map: execute→run, consult→ask, deny→block.

Retain and narrow

  1. Retain the existing sidecar as a descriptive joint option-wording perturbation, outside H1/H2. It is not a sixth primary rung or a meaning-preserving control. No vocabulary replacement or apparatus rebuild is required by this disposition.
  2. No class supports a semantic-invariance claim. Ask-gold results remain uninterpretable as meaning-preserving label sensitivity, but may be reported descriptively under the existing validity requirements. consult permits a non-human/resource-consultation reading. Its presence as a competing choice can also affect run- and block-gold choices: excluding ask-gold rows does not remove the semantic confound. Do not attribute a gold class's flips to its corresponding alias alone.
  3. Preserve raw and inverse-mapped outputs privately, alongside the existing pooled and all-gold-class flip summaries, with their provenance. This is a retention requirement, not permission to publish private/raw traces. Preserve failures, partial and superseded findings; do not silently replace them.
  4. Character-length differences are not measured token differences. The existing perturbation changes wording and length jointly; flips do not identify a semantics-only effect. Adapter-exposed probabilities are not automatically logits. Unavailable option distributions, including Qwen's under its historical output contract, remain NOT MEASURED, not zero.
  5. Keep semantic_equivalence_independently_approved: false. This disposition does not establish semantic equivalence or target-host acceptance. Preserve the existing paired-validity and no-imputation rules: missing or invalid paired observations yield no sidecar conclusion, not a partial favorable result.
  6. Do not invent implemented observables. The inspected scorer reports pooled and gold-stratified flip counts, not a transition matrix. No new metric, transition-count implementation or numerical re-analysis is added or required here.

Existing evidence and open gates

Vesper's #91 review 192 is bound to 4472695d095d7d7665f8df1a098cf88ff5d09964; her #89 review 203 is a COMMENT bound to 3cc5dbbbb77eebbb50efcd0cc37bacac0b52e619; her #90 comment 2059 records the alias-meaning objection. Retain all three. This addendum resolves the question-owner interpretation choice by retaining the vocabulary without a semantic-invariance claim for any class; it does not rewrite those reviews or claim a new independent approval.

#89 remains a non-dispatchable intermediate: its criteria-deleted R4 must not enter a run. #91 preserves the corrected ordered canonical label→null-description shape, supported at the pinned source level, not measured target-host acceptance. Actual consuming-adapter verification, LM tokenizer traces/length-match obligations, runtime acceptance, model identity/environment, device admission and the existing independent-review/freeze requirements remain unresolved where applicable. Source-level review does not close them.

Scope: append-only documentation on existing #90/#91 only. No code, payload, manifest, branch, PR body, review verdict, issue state or scientific ledger is changed. No new review dispatch, protocol freeze, merge, force-push, deletion, access change, experiment, model run/load/download, compute reservation/spend, private/raw trace publication, or reopening of execution gates. Existing capability-envelope policy is unchanged; no repeated per-run operator-approval requirement is introduced. Marathon inclusion remains tentative and outside this disposition.

## 2026-10-04 disposition — retain the sidecar; narrow its interpretation Recorded by **Pyrrha**, question owner, on Cassie's direct documentation-only instruction. This dated addendum retains earlier comments and approvals rather than rewriting or dismissing them. **Exact revision binding:** - PR #91: `4472695d095d7d7665f8df1a098cf88ff5d09964`. - Base PR #89: `3cc5dbbbb77eebbb50efcd0cc37bacac0b52e619`. - Existing map: `run→execute`, `ask→consult`, `block→deny`. - Inverse output map: `execute→run`, `consult→ask`, `deny→block`. ### Retain and narrow 1. **Retain the existing sidecar as a descriptive joint option-wording perturbation, outside H1/H2.** It is not a sixth primary rung or a meaning-preserving control. No vocabulary replacement or apparatus rebuild is required by this disposition. 2. **No class supports a semantic-invariance claim.** Ask-gold results remain uninterpretable *as meaning-preserving label sensitivity*, but may be reported descriptively under the existing validity requirements. `consult` permits a non-human/resource-consultation reading. Its presence as a competing choice can also affect run- and block-gold choices: excluding ask-gold rows does not remove the semantic confound. Do not attribute a gold class's flips to its corresponding alias alone. 3. **Preserve raw and inverse-mapped outputs privately**, alongside the existing pooled and all-gold-class flip summaries, with their provenance. This is a retention requirement, not permission to publish private/raw traces. Preserve failures, partial and superseded findings; do not silently replace them. 4. **Character-length differences are not measured token differences.** The existing perturbation changes wording and length jointly; flips do not identify a semantics-only effect. Adapter-exposed probabilities are not automatically logits. Unavailable option distributions, including Qwen's under its historical output contract, remain **NOT MEASURED**, not zero. 5. **Keep `semantic_equivalence_independently_approved: false`.** This disposition does not establish semantic equivalence or target-host acceptance. Preserve the existing paired-validity and no-imputation rules: missing or invalid paired observations yield no sidecar conclusion, not a partial favorable result. 6. **Do not invent implemented observables.** The inspected scorer reports pooled and gold-stratified flip counts, not a transition matrix. No new metric, transition-count implementation or numerical re-analysis is added or required here. ### Existing evidence and open gates Vesper's #91 review 192 is bound to `4472695d095d7d7665f8df1a098cf88ff5d09964`; her #89 review 203 is a `COMMENT` bound to `3cc5dbbbb77eebbb50efcd0cc37bacac0b52e619`; her #90 comment [2059](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/issues/90#issuecomment-2059) records the alias-meaning objection. Retain all three. This addendum resolves the question-owner interpretation choice by retaining the vocabulary **without a semantic-invariance claim for any class**; it does not rewrite those reviews or claim a new independent approval. #89 remains a non-dispatchable intermediate: its criteria-deleted R4 must not enter a run. #91 preserves the corrected ordered canonical label→null-description shape, supported at the pinned source level, **not measured target-host acceptance**. Actual consuming-adapter verification, LM tokenizer traces/length-match obligations, runtime acceptance, model identity/environment, device admission and the existing independent-review/freeze requirements remain unresolved where applicable. Source-level review does not close them. **Scope:** append-only documentation on existing #90/#91 only. No code, payload, manifest, branch, PR body, review verdict, issue state or scientific ledger is changed. No new review dispatch, protocol freeze, merge, force-push, deletion, access change, experiment, model run/load/download, compute reservation/spend, private/raw trace publication, or reopening of execution gates. Existing capability-envelope policy is unchanged; no repeated per-run operator-approval requirement is introduced. Marathon inclusion remains tentative and outside this disposition.
Owner

Closing: disposition recorded 2026-10-04; PR91 content preserved on main via #119 (a5b5e927). No dispatch pending.

Closing: disposition recorded 2026-10-04; PR91 content preserved on main via #119 (a5b5e927). No dispatch pending.
Sign in to join this conversation.
No labels
R03
research-b
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany#90
No description provided.