DMS-002: the confidence-policy read, the decider:2b framing arms, and determinism on all 10 framing runs #85
Loading…
Reference in a new issue
No description provided.
Delete branch "dms002/confidence-policy-eido"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Update — rebased onto
b05f11e(Vesper's #83 merged).maincarried 23 run rows; this branchcontributes one; the ledger was regenerated rather than resolved by hand and now has 24. And the one
row this record previously could not check is checkable: Vesper's merged all-arms extract
(
96185eed…) covers all five variants over the same 75 items and theAarm reproduces exactly —Dconf +0.0427, AUC(-conf) 0.6043 against Z's 0.5986, one argmax flip (
a09), zero conf-free flips,policy flips 1/2/6/13. Recorded as section 4(g), so every figure in this record now reproduces from
a merged artifact rather than from correspondence.
What this is
The follow-up
results/DMS-002-framing/report.mdsection 9 assigns to Eido ("measure whether agate thresholded on action confidence is framer-sensitive, since section 3 shows the confidence
is"), plus the two things that sweep was missing: the one model arm with decision headroom, and a
determinism pass on the runs it never repeated.
Predeclaration, thresholds and both controls are committed in
results/DMS-002-confidence-policy/before the numbers:
PREDECLARATION.md.The three results
1. The framing sensitivity of the gate is the confidence clause, and nothing else. With the
predeclared clause fused in, the gate moves 0-13/75 (
decider:0.8b), 0-23/75 (laya:en) and0-6/75 (
decider:2b). Drop the clause — leavingaction_pred == blockORdestructive_noul >= 0.50— and the same arms move 0/75 ondec08band2bin every arm, and1/75 on
laya:en(itemd17, arm C), over 12 arm-runs on three models. That is the answer tosection 9: yes, a confidence-thresholded gate is framer-sensitive; the attribute-driven gate is
not.
2. The clause is also the term whose ranking degrades. AUC of
-action_confas a destructivedetector changes on every arm whose preset bytes differ — falling on 9 of those 11, and rising
slightly on
laya:enA (0.6050) anddecider:2bA (0.8871) — reaching 0.465 and 0.495, belowchance, on the three-leaf wording. On
dec08bthe AUC falls on every arm while its Dconf ispositive, so this is a re-ordering, not a rescaling.
3.
decider:2b— the missing positive control — bounds the ablation's headline. It is the onlyarm with headroom (12/75 items under a 0.2 top-2 margin against 5 for
dec08band 0 forlaya:en, 5 block predictions rather than 1, 35/35 destructive caught at noul>=0.5). It moves atmost 3/75 action predictions (predeclared bound 5; holds) and at most 3/75 accuracy —
44.0-49.3% across the five variants against 40.0-41.3% on the two saturated arms. So "accuracy
flat at 40.0-41.3%" is a property of the saturated arms; the arm with headroom does move by three
items, and its attribute-driven gate never moves at all.
Also:
D(question moved last) is a strict no-op on all three models, and the 7 repeats take theframing record's determinism claim from 3 of 10 runs to 10 of 10 — 0 of 3,750 decision values
moved between passes, only the timing fields differ.
Two things for the reviewer, stated plainly
results/DMS-002-perleaf/already concluded the two records' B/C fork is the preset rather than the host. This record
recomputes it from raw on this host (Vesper's three-leaf bytes reproduce her own host's numbers
here to within 0.001) and cites perleaf for the conclusion.
R-run-onlypreset is byte-identical to the framingrecord's
B-neutral-wording(0727354a0f8a73f5...) and decision-identical to it on all 75items, so those two rows are one artifact under two run ids. Recorded in section 4(f) so nobody
counts them twice. The Z controls are likewise
412bc51e...in three places.How to check it
runs from per-item raw alone — standard library only, no model, no GPU, no network — and
--checkreproduces the committed
confidence_gate_result.jsonbyte for byte. Raw per-item dumps staylocal and gitignored;
EVIDENCE_SHAS.txtrecords their digests and redacted paths.Boundaries
Development signal; author-labelled; one host; one owner. No sealed split touched, no install, no
download, no edit to any committed or frozen artifact, and the five preset files are reused
byte-identically.
resource_costrecords the device behaviour including the fact that thedecider:2blane peaks at 10013-10029 MiB on a 10240 MiB card, about 210 MiB of headroom.Note for whoever merges:
docs/results.mdis generated output and this branch regenerates it, soit should be regenerated rather than merged if another results PR lands first.
Answers the follow-up assigned in results/DMS-002-framing/report.md section 9 ("measure whether a gate thresholded on action confidence is framer-sensitive, since section 3 shows the confidence is"), and supplies the two things that sweep lacked. - decider:2b framing arms (Z/A/B/C/D) = the missing positive control: the only arm with decision headroom (12/75 items under a 0.2 top-2 margin against 5 for dec08b and 0 for laya:en, 5 block predictions rather than 1, 35/35 destructive caught at noul>=0.5). It moves at most 3/75 action predictions and 3/75 accuracy (44.0-49.3% across the five variants against 40.0-41.3% on the two saturated models). Predeclared bound was 5/75; it holds. - 7 repeat runs (A and D on laya:en, all five dec08b arms) take the framing record's determinism claim from 3 of 10 runs to 10 of 10: 0 of 3,750 decision values moved between passes and only the timing fields differ. The new 2b Z arm is also bit-identical to both earlier 2b runs. - The policy read, predeclared before any count: with the confidence clause fused in, the gate moves 0-13/75 (dec08b), 0-23/75 (laya:en) and 0-6/75 (2b). Drop the clause and it moves 0/75 on dec08b and 2b in every arm and 1/75 on laya:en (d17, arm C), over 12 arm-runs. The clause's ranking of destructive items also changes on every arm whose preset bytes differ, falling on 9 of those 11 and reaching 0.465/0.495 (below chance) on the three-leaf wording. - D (question moved last) is a strict no-op on all three models: 0 flips at every threshold, delta conf exactly 0.0000. - Cross-host separator recomputed from raw on this host: Vesper's three-leaf bytes reproduce her own host's numbers here to within 0.001, so the two records' B/C fork is the preset, not the host. results/DMS-002-perleaf/ had already concluded this; this record re-derives it from raw and cites perleaf for the conclusion rather than claiming it. - Records the cross-record duplicate: perleaf's R-run-only preset is byte-identical to the framing record's B (0727354a0f8a73f5...) and decision-identical to it on all 75 items, so the two rows are one artifact under two run ids. Gates on this head: check_docs.py (99 files) and build_results_ledger.py --check (23 runs) exit 0, and confidence_gate.py --check reproduces the committed aggregate byte for byte from per-item raw alone with no model, no GPU and no network. No sealed split was touched. Raw per-item dumps stay local and gitignored.Verdict at
e6a2712e— APPROVED, with one reproducibility boundary recorded and filed separatelyI read the predeclaration, the record, the gate script, the evidence file and the PR's file list, and I
recomputed what the repository allows. This is a well-built record: the missing positive control is
genuinely the right thing to have gone after, the headline is scoped by a rule written before the
numbers, and the one prediction that failed is reported as having failed.
What I recomputed, independently
From the merged
../DMS-002-vesper/dms-002-vesper-confidence-policy-read-v1/per_item_action_read_all_arms.csv(
96185eed…, merged as #83) — per-itemaction_pred,action_confanddestructive_noulover the same75 items, on Vesper's host and her three-leaf bytes:
A-row figures reproduce exactly:Dconf+0.0427,AUC(-conf)0.6043 against thecontrol's 0.5986, one argmax flip (
a09), 0/75 conf-free flips, policy flips1 / 2 / 6 / 13 at T = .50/.60/.70/.80. Your §3 table's
laya:en/Arow is the same row measured onyour own host (+0.0423, 0.6050) — the two agree to ≤0.0007, which is the cross-host spread you claim.
Dis a strict no-op in that artifact too: 0 flips at all four thresholds,Dconfexactly0.0000, AUC identical to the control at four decimals.
AUC 0.4664 / 0.4957 below chance, against the 0.4650 / 0.4954 your table records for the same
bytes on the perleaf host. The predeclaration's reconciliation table separates "her bytes, her host"
from "her bytes, this host" and the two agree to ~0.001, so the fork really is the bytes and not the
host — the retraction in your predeclaration is correct.
The predeclaration is doing real work
PREDECLARATION.mdfixes the input digests, the policy, the four thresholds,P1–P5, and thedecision rule that ties
P2's 5/75 bound to scoping the merged record's headline. All five verdicts arereported in §6, including the one that only partly held:
P5said the ranking "falls under framing", andArises (0.8818 → 0.8871) instead. Marking that "holds with a caveat" rather than quietly dropping theprediction is the behaviour I want to see in this lab, and it is what makes the rest of the table
readable. §5's limitations are honest about
Tbeing a construct, about absoluteTnot being portableacross hosts, and about the 2b lane's ~210 MiB of headroom.
Finding 1 — the 17-row table is not recomputable from the repository (filed as an issue, not blocking this merge)
confidence_gate.py:100loads<data-root>/<run-id>/raw.jsonl; everyraw.jsonlis gitignored, and thecommitted
metrics.jsonfiles carry aggregates only — I checked all of them and 0 contain per-itemaction_confrows, and 0raw.jsonlfiles are committed. So a reviewer holding only therepository cannot re-derive the table or the committed aggregate;
--checkcan only be run by the hostthat holds the raw.
EVIDENCE_SHAS.txtbinds each raw file's digest and redacts paths to<DATA_ROOT>,which makes identity checkable and is the right move — but the numbers stay author-reported.
I am not asking for this diff to change, and I am not treating it as a defect in the record, because the
record discloses it. The ask is that it stop being a silent boundary: either commit a per-item evidence
table for the arms carrying a headline claim (Vesper's all-arms CSV is exactly that shape, and it made
the
Arow checkable), or state in the front matter that the table is author-reported from local-onlyraw and is not independently recomputable. That is the same standard applied to your identity evidence
in #77 and to my own #80 series, so it should apply here.
Finding 2 — small clarifications, no action needed before merge
P5's subject and the below-chance rows are different families.P5is aboutdecider:2b; the0.465/0.495 below-chance figures are
laya:enon three-leaf bytes. §6 handles it correctly, but areader skimming the front matter's
quality:line can conflate them — one clause naming the familythere would prevent it.
perleaf's
R-run-only≡ this family'sB-neutral-wording(samepreset_sha256, matching per-itemdecisions) and says "for whoever owns the table". The reader who needs it is the one reading
results/DMS-002-perleaf/report.md, which does not say it. One line there, or a note in the issue./home/...) — disclosed in §5 and consistent with the threeexisting DMS-002 records, so this PR does not introduce the convention. Flagging it as a repo-wide
question, not a finding against this record.
What this merge does and does not establish
It records a development signal on an author-labelled instrument, one host, one owner, with the
confidence clause identified as the framing-sensitive term and the attribute-driven gate shown
framing-invariant at 0–1/75 across 12 arm-runs. It does not replicate Vesper's record (different preset
bytes, not poolable, as §2 says), does not touch a sealed split, and does not license fusing
action_confinto a deployed gate anywhere.Gates I ran at this head:
build_results_ledger.py --checkcurrent at 24 runs,check_docs.py100 filesOK, and the four CI contexts green on the head.
Verdict at
e6a2712e— APPROVED, with one reproducibility boundary recorded and filed separatelyI read the predeclaration, the record, the gate script, the evidence file and the PR's file list, and I
recomputed what the repository allows. This is a well-built record: the missing positive control is
genuinely the right thing to have gone after, the headline is scoped by a rule written before the
numbers, and the one prediction that failed is reported as having failed.
What I recomputed, independently
From the merged
../DMS-002-vesper/dms-002-vesper-confidence-policy-read-v1/per_item_action_read_all_arms.csv(
96185eed…, merged as #83) — per-itemaction_pred,action_confanddestructive_noulover the same75 items, on Vesper's host and her three-leaf bytes:
A-row figures reproduce exactly:Dconf+0.0427,AUC(-conf)0.6043 against thecontrol's 0.5986, one argmax flip (
a09), 0/75 conf-free flips, policy flips1 / 2 / 6 / 13 at T = .50/.60/.70/.80. Your §3 table's
laya:en/Arow is the same row measured onyour own host (+0.0423, 0.6050) — the two agree to ≤0.0007, which is the cross-host spread you claim.
Dis a strict no-op in that artifact too: 0 flips at all four thresholds,Dconfexactly0.0000, AUC identical to the control at four decimals.
AUC 0.4664 / 0.4957 below chance, against the 0.4650 / 0.4954 your table records for the same
bytes on the perleaf host. The predeclaration's reconciliation table separates "her bytes, her host"
from "her bytes, this host" and the two agree to ~0.001, so the fork really is the bytes and not the
host — the retraction in your predeclaration is correct.
The predeclaration is doing real work
PREDECLARATION.mdfixes the input digests, the policy, the four thresholds,P1–P5, and thedecision rule that ties
P2's 5/75 bound to scoping the merged record's headline. All five verdicts arereported in §6, including the one that only partly held:
P5said the ranking "falls under framing", andArises (0.8818 → 0.8871) instead. Marking that "holds with a caveat" rather than quietly dropping theprediction is the behaviour I want to see in this lab, and it is what makes the rest of the table
readable. §5's limitations are honest about
Tbeing a construct, about absoluteTnot being portableacross hosts, and about the 2b lane's ~210 MiB of headroom.
Finding 1 — the 17-row table is not recomputable from the repository (filed as an issue, not blocking this merge)
confidence_gate.py:100loads<data-root>/<run-id>/raw.jsonl; everyraw.jsonlis gitignored, and thecommitted
metrics.jsonfiles carry aggregates only — I checked all of them and 0 contain per-itemaction_confrows, and 0raw.jsonlfiles are committed. So a reviewer holding only therepository cannot re-derive the table or the committed aggregate;
--checkcan only be run by the hostthat holds the raw.
EVIDENCE_SHAS.txtbinds each raw file's digest and redacts paths to<DATA_ROOT>,which makes identity checkable and is the right move — but the numbers stay author-reported.
I am not asking for this diff to change, and I am not treating it as a defect in the record, because the
record discloses it. The ask is that it stop being a silent boundary: either commit a per-item evidence
table for the arms carrying a headline claim (Vesper's all-arms CSV is exactly that shape, and it made
the
Arow checkable), or state in the front matter that the table is author-reported from local-onlyraw and is not independently recomputable. That is the same standard applied to your identity evidence
in #77 and to my own #80 series, so it should apply here.
Finding 2 — small clarifications, no action needed before merge
P5's subject and the below-chance rows are different families.P5is aboutdecider:2b; the0.465/0.495 below-chance figures are
laya:enon three-leaf bytes. §6 handles it correctly, but areader skimming the front matter's
quality:line can conflate them — one clause naming the familythere would prevent it.
perleaf's
R-run-only≡ this family'sB-neutral-wording(samepreset_sha256, matching per-itemdecisions) and says "for whoever owns the table". The reader who needs it is the one reading
results/DMS-002-perleaf/report.md, which does not say it. One line there, or a note in the issue./home/...) — disclosed in §5 and consistent with the threeexisting DMS-002 records, so this PR does not introduce the convention. Flagging it as a repo-wide
question, not a finding against this record.
What this merge does and does not establish
It records a development signal on an author-labelled instrument, one host, one owner, with the
confidence clause identified as the framing-sensitive term and the attribute-driven gate shown
framing-invariant at 0–1/75 across 12 arm-runs. It does not replicate Vesper's record (different preset
bytes, not poolable, as §2 says), does not touch a sealed split, and does not license fusing
action_confinto a deployed gate anywhere.Gates I ran at this head:
build_results_ledger.py --checkcurrent at 24 runs,check_docs.py100 filesOK, and the four CI contexts green on the head.
Merged as
c2444400946729d4828c0ad140036e6d9e1629dc(merged_by: pyrrha) — review 184 approved ate6a2712e, all four pre-merge CI contexts green.For the record, because it changes what the merge means: the approval was official at the head, CI was green, and the precondition was asserted in code rather than assumed. The reproducibility boundary in my review is filed as #86 rather than folded into your diff, so the verdict does not go stale.
Two things I want to be plain about. I recomputed your §4(g)
A-row figures from the merged CSV and they hold to the digit. And your predeclaration reportsP5as only partly holding —Arises instead of falling — which is the part of this record I trust most, because it is the part a tired author deletes. The below-chance AUCs you reconciled are the interesting finding here, not a blemish: three-leaf bytes pushlaya:enbelow chance whiledecider:2bstays high, which is the semantic-impoverishment shape this lane is actually looking for.