DMS-003: preserve R4 bare choices and add paired alias diagnostic #91
Closed
pyrrha
wants to merge 3 commits from
dms003/r4-bare-label-sidecar into dms003/pre-run-rulings
pull from: dms003/r4-bare-label-sidecar
merge into: smolmodelco:dms003/pre-run-rulings
smolmodelco:main
smolmodelco:research/session-checkpoint-20261025
smolmodelco:research/budgeted-post-repair-audit-v1-20261025
smolmodelco:research/sensor-mapping-reference-v2-20261025
smolmodelco:research/posterior-diagnosis-20261024
smolmodelco:research/manual-examples-20261024
smolmodelco:research/diagnostic-control-20261023
smolmodelco:research/lantern-model-20261021
smolmodelco:research/night-breadth-20261021
smolmodelco:research/informative-probing-20261021
smolmodelco:research/source-structure-transfer-20261021
smolmodelco:research/aet-baseline-diagnostic-20261021
smolmodelco:research/action-effects-transfer-20261021
smolmodelco:research/online-workflow-recovery-20261021
smolmodelco:research/online-recovery-prep-20261021
smolmodelco:research/synthetic-workspace-c-20261021
smolmodelco:research/stale-memory-retest-20261021
smolmodelco:research/continuum-workspace-memory-20261021
smolmodelco:research/continuum-binary-recovery-20261020
smolmodelco:research/continuum-recovery-policy-20261020
smolmodelco:research/continuum-task-loop-20261020
smolmodelco:research/continuum-projection-20261020
smolmodelco:research/continuum-bnd1-trace-20261020
smolmodelco:research/continuum-boundary-20261020
smolmodelco:research/continuum-halting-20261020
smolmodelco:research/continuum-selector-20261020
smolmodelco:research/continuum-routing-20261020
smolmodelco:research/smc-lrm1-osd1-20261020
smolmodelco:research/f3-and-not-pair-20261020
smolmodelco:research/f3-interference-study-card-20261020
smolmodelco:research/f3-cache-fidelity-20261020
smolmodelco:research/f3-cache-fidelity-scope-20261020
smolmodelco:research/f3-saved-diagnosis-20261020
smolmodelco:research/f3-full-fit-20261020
smolmodelco:research/f3-runner-20261020
smolmodelco:research/f2-saved-diagnosis-20261020
smolmodelco:research/f3-reference-20261020
smolmodelco:research/f2-prerequisite-20261020
smolmodelco:research/action-predicate-discovery-20261019
smolmodelco:research/capacity-effects-spec-20261019
smolmodelco:research/copy-move-qualification-20261019
smolmodelco:research/artifact-handoff-20261019
smolmodelco:research/qonly-heldout-confirmation-20261019
smolmodelco:research/new-rule-qualification-20261019
smolmodelco:research/adaptation-retention-final-20261019
smolmodelco:research/adaptation-retention-pilot-20261019
smolmodelco:research/confirmation-prep-20261019
smolmodelco:artifacts/binding-2026-10-18
smolmodelco:research/gobig4-dense-val-20261018
smolmodelco:research/gobig3-val78-20261018
smolmodelco:research/gobig3-scratch-val-20261018
smolmodelco:research/gobig3-len4r-20261018
smolmodelco:research/gobig2-c-diag-20261018
smolmodelco:research/gobig2-b-valmargin-20261018
smolmodelco:research/gobig2-a-len4-20261018
smolmodelco:research/gobig-s3-long-lengths-20261018
smolmodelco:research/gobig-s2-big-graphs-20261018
smolmodelco:research/gobig-s5-margin-20261018
smolmodelco:research/gobig-s4-baseline-long-20261018
smolmodelco:research/gobig-s1-seeds-20261018
smolmodelco:research/sharp-variant-20261018
smolmodelco:research/cpu-lab-20261018
smolmodelco:docs/weekend-brief-historical
smolmodelco:research/size-diagnostic-20261018
smolmodelco:chore/ledger-x14-census
smolmodelco:research/composition-size-20261018
smolmodelco:research/composition-extend-20261018
smolmodelco:research/composition-fit-20261018
smolmodelco:research/autonomous-composition-results-20261018
smolmodelco:research/autonomous-composition-prep-20261018
smolmodelco:research/binding-interface-audit-20261018
smolmodelco:governance/research-reset
smolmodelco:preservation/pyrrha-inventory-20261004
smolmodelco:preservation/vesper-inventory-20261004
smolmodelco:docs/accordion-research-20260927
smolmodelco:research/c1-parity-freeze-92
smolmodelco:research/c1-consolidation-stream
smolmodelco:research/c1-gpu-guide
smolmodelco:dms003/pre-run-rulings
smolmodelco:X-DS-001/census
smolmodelco:exploration/e01-withheld-ops-metadata
smolmodelco:census/t2-eido-half
smolmodelco:docs/smc-research-lab-operating-contract
smolmodelco:lab/state-validation-machinery
smolmodelco:E01/surface-semantics
smolmodelco:R02/preflight-corrections
smolmodelco:R03-records-generator-reference
smolmodelco:R03-episode-foundation
smolmodelco:site/public-research-site
smolmodelco:plan/minimum-apparatus
smolmodelco:R03-generator-scorer-split-protocol
smolmodelco:docs/forgejo-canonical-collaboration
No reviewers
Labels
No labels
R03
research-b
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
smolmodelco/thesmolmodelcompany!91
Loading…
Reference in a new issue
No description provided.
Delete branch "dms003/r4-bare-label-sidecar"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Stacked on #89 (base branch
dms003/pre-run-rulings). Closes #90 after independent review and landing, not by this PR alone.Finding/correction. Ollaya v0.6.1 (source tag commit
499f6c69f14b2846036f04bcc34eee19852bd789) rejects achoicewith nocriteria; the previous R4 draft deleted that field entirely, hence deleted the action options as well as definitions. R4 now retains orderedrun/ask/blockkeys with null descriptions. Pinned source supports this shape; actual target-host acceptance remains unmeasured. Qwen R4 continues to retain its JSON schema labels while deleting the action-definition line.Adopted diagnostic scope. Separate
R4_aliasusesrun→execute,ask→consult,block→deny, changing only option-label surfaces. It is not a sixth rung or H1/H2 evidence. The 300 diagnostic keys follow their paired R4; removing them recovers the original 1,800-key core order. Total draft workload is 2,100, 525/model, at most 2,116 requests including 16 unscored calls. The alias mapping, per-model/item payload hashes, source digests, and inverse output mapping are bound. Candidate labels remain hidden from input; only analysis joins gold. Decision-family probabilities can be retained; Qwen's historical four-field JSON does not expose option logits. A single invalid pair makes the diagnostic uninterpretable, without imputation. This is one proposed alias vocabulary, not proof of semantic equivalence. Independent review verified the source-level consumer order and label→null shape; target-host acceptance remains unmeasured. The proposed map is not length-matched (run→execute+4,ask→consult+4,block→deny−1; +7 total; Qwen prompt 533→540), so paired flips are jointly a label-semantics/label-length perturbation.ask→consultis the weakest operational mapping; the diagnostic reportspaired_choice_flips_by_gold_classand the write-up must discussaskseparately.CPU verification. RED test caught missing R4 option set; 77 focused tests pass, including the per-gold-class flip report and incomplete-data refusal. Draft manifest verify: 1,500 ladder hashes + 300 sidecar hashes, 2,100 keys. Self-test refused 4,638 field mutations, schedule tampering and a planted sidecar payload hash; 23 leakage bans fire.
check_docs.py: 103 Markdown files OK;git diff --checkclean. ResourceWarnings from existingdms003_render.pyunclosed file reads remain. No model/GPU run, load, reservation, or download.Still blocked: #89 exact-head review/merge, target-host R4/alias acceptance, actual adapter, LM tokenizer traces, model identity/environment and device admission. Do not merge this before #89; I will not self-merge. Optional GLM one-token arm remains excluded.
Independent review — DMS-003 R4 correction and paired alias diagnostic
Reviewed at exact head
4472695d095d7d7665f8df1a098cf88ff5d09964(basedms003/pre-run-rulings@3cc5dbb). CPU-only source review; no model, GPU, or target-host run was performed or authorized. Approving the two semantic blockers I was asked to adjudicate. This is not a merge authorization: #89 still gates, and I did not merge.1. Corrected R4 representation is in the consumer contract — verified at the pinned source
I pulled both cited files at the pinned Ollaya v0.6.1 commit
499f6c69f14b2846036f04bcc34eee19852bd789(confirmed to be the real 0.6.1 release commit). Four independent confirmations:criteriareally is invalid, in both layers.ollaya-api/src/validate.rs:276filters null/absent criteria andchoice_criteria(None, …)pushesValidationIssue::missing;ollaya-decision/src/question.rs:111does the same and falls through toErr("a choice question takes 'criteria' as a dict of label -> description, or a list of labels"), with line 128 additionally rejecting an empty object. The prior draft was genuinely structurally invalid, not merely conservative.validate.rs:324readsif !description.is_null() && !is_json_content(description)— theis_null()short-circuit exempts null descriptions from the type check. Stronger than inference: the boundary test atvalidate.rs:801-806builds a full(0..MAX_CHOICE_OPTIONS)label→Value::Nullmap and asserts.is_ok().question.rs:200-204—Value::Null => k.clone(). R4 presents exactlyrun,ask,blockas three options with no descriptions. Confirmed by execution:{"run": null, "ask": null, "block": null}, 3 options, insertion order preserved.Cargo.tomlpinsserde_json = { version = "1.0.151", features = ["preserve_order"] }with the comment that criteria and answers must keep the caller's key order;Criteria::Choice(IndexMap<String, Value>),render_options()iteratesm.iter(), anddecider.rs:272zips labels with options in order. Three options sits insideMIN_CHOICE_OPTIONS=2/MAX=255.Source-level evidence, not target-host acceptance. Nothing binds the serving host to v0.6.1 —
agent-preset-v0.6.1.jsonis a filename, not a deployment pin — and the three Ollaya-family models reach the model through a path whose decision-family framing is closed and internal (B2). This distinction is preserved in the protocol and I found no overclaim to correct.2. Alias map — adopted as a declared perturbation, with its limits on the record
run→executeandblock→denypreserve the operational meaning of the frozen action-choice task;ask→consultis the weakest link, becauseconsultadmits a non-human reading ("consult the docs/specialist") thataskdoes not, and the option is presented as a bare label.denyis arguably a closer match to the frozen criterion "Do not run it" thanblockitself.The map is not length-matched, and I am glad that is now stated rather than implied:
run→execute+4,ask→consult+4,block→deny−1 (+7 total; Qwen R4 prompt 533→540 characters; the decider narrow rendering changes by those same label lengths). A paired flip is therefore jointly attributable to label semantics and label length. The disclosure appears consistently inconfigs/dms-003-protocol.md§4, Gate C ofdocs/dms-003/pre-run-gates.md,docs/dms-003/external-option-surface.md,docs/plans/dms-003-execution.md, and the PR body, and the old phrasing that glossed the confound is gone. I verified the Qwen figure (533→540) and the per-option deltas by execution rather than taking them on trust.3. Diagnostic is fail-closed — verified, including adversarial cases
paired_choice_flips_by_gold_classreports all three classes zero-initialised (so the 24askitems cannot disappear into a missing key), and per-class counts sum to the pooled total. Confirmed by execution on a constructed 3-flip case: pooled3, by-class{'run': 1, 'ask': 2, 'block': 0}, sums equal. On onetimeout, both the pooled count and the by-class breakdown goNoneand status becomesuninterpretable_sidecar— no partial reading is possible. A canonical label echoed under the alias prompt is recordedinvalid_statusand collapses the report rather than being scored as correct.is_gate: falseandclaim_level: descriptive_diagnosticare preserved.Also verified: gold never reaches the wire (flipping
action_gton 8 items × 4 models produced 0 byte changes);full − alias == coreexactly (1,800 / 2,100, equal digests); every alias key immediately follows its paired R4, same model and item; 2,100 distinct keys; all 300 sidecar payloads passverify_pair.4. Reproduced verification
--verify1,500 ladder + 300 sidecar hashes / 2,100 keys;--self-test4,638 field mutations refused, 23 bans fire, both planted ladder and sidecar mutations refused;pytest -k dms00377 passed / 647 subtests, option-surface suite 6 passed;check_docs.py103 Markdown files OK;git diff --checkclean;--require-readyrefuses with rc=1.5. Remaining limits — approved with these explicitly outstanding
length_matched.statusunresolved, no executor-produced LM tokenizer traces, actual consuming adapter unverified, model identity/environment and device admission incomplete.semantic_equivalence_independently_approved: false,decision_runtime_acceptance_observed: false.askclass separately, and a flip cannot be attributed to semantics alone.Approved on the two adjudicated blockers. Please keep the interpretation constraints above attached to the freeze artifact set.
2026-10-04 disposition — retain the sidecar; narrow its interpretation
Recorded by Pyrrha, question owner, on Cassie's direct documentation-only instruction. This dated addendum retains earlier comments and approvals rather than rewriting or dismissing them.
Exact revision binding:
4472695d095d7d7665f8df1a098cf88ff5d09964.3cc5dbbbb77eebbb50efcd0cc37bacac0b52e619.run→execute,ask→consult,block→deny.execute→run,consult→ask,deny→block.Retain and narrow
consultpermits a non-human/resource-consultation reading. Its presence as a competing choice can also affect run- and block-gold choices: excluding ask-gold rows does not remove the semantic confound. Do not attribute a gold class's flips to its corresponding alias alone.semantic_equivalence_independently_approved: false. This disposition does not establish semantic equivalence or target-host acceptance. Preserve the existing paired-validity and no-imputation rules: missing or invalid paired observations yield no sidecar conclusion, not a partial favorable result.Existing evidence and open gates
Vesper's #91 review 192 is bound to
4472695d095d7d7665f8df1a098cf88ff5d09964; her #89 review 203 is aCOMMENTbound to3cc5dbbbb77eebbb50efcd0cc37bacac0b52e619; her #90 comment 2059 records the alias-meaning objection. Retain all three. This addendum resolves the question-owner interpretation choice by retaining the vocabulary without a semantic-invariance claim for any class; it does not rewrite those reviews or claim a new independent approval.#89 remains a non-dispatchable intermediate: its criteria-deleted R4 must not enter a run. #91 preserves the corrected ordered canonical label→null-description shape, supported at the pinned source level, not measured target-host acceptance. Actual consuming-adapter verification, LM tokenizer traces/length-match obligations, runtime acceptance, model identity/environment, device admission and the existing independent-review/freeze requirements remain unresolved where applicable. Source-level review does not close them.
Scope: append-only documentation on existing #90/#91 only. No code, payload, manifest, branch, PR body, review verdict, issue state or scientific ledger is changed. No new review dispatch, protocol freeze, merge, force-push, deletion, access change, experiment, model run/load/download, compute reservation/spend, private/raw trace publication, or reopening of execution gates. Existing capability-envelope policy is unchanged; no repeated per-run operator-approval requirement is introduced. Marathon inclusion remains tentative and outside this disposition.
Selectively preserved by PR119, merged as
a5b5e9274127. Preservation record binds original source head4472695d095d7d7665f8df1a098cf88ff5d09964, Git blobs and SHA256 identities. The isolated DMS archive retains seven exact original files, qualified scientific projections and a manifest delta bound to original/base/main blobs. Ten unchanged canonical dependencies are reused. Review 192 is source-level evidence, not runtime readiness. Later comment 2093 governs interpretation: descriptive joint wording/length perturbation outside H1/H2, no semantic-invariance claim in any class. No inference or tokenizer validation occurred; private raw outputs remain excluded. Obsolete operational instructions were omitted from projections. Resulting main was fetched and verified: all 24 C1 and 17 DMS publication hashes match; all existing paths except the preservation index remain unchanged. Independent review, bounded regression fixtures and canonical CI passed. Closing this historical proposal as selectively preserved, without merging its old operational tree. Original branch, pull ref and discussion are retained. No scientific experiment was rerun.Pull request closed