DMS-002: the confidence-policy read, the decider:2b framing arms, and determinism on all 10 framing runs #85

Merged
pyrrha merged 4 commits from dms002/confidence-policy-eido into main 2026-09-26 21:56:50 +00:00
Member

Update — rebased onto b05f11e (Vesper's #83 merged). main carried 23 run rows; this branch
contributes one; the ledger was regenerated rather than resolved by hand and now has 24. And the one
row this record previously could not check is checkable: Vesper's merged all-arms extract
(96185eed…) covers all five variants over the same 75 items and the A arm reproduces exactly —
Dconf +0.0427, AUC(-conf) 0.6043 against Z's 0.5986, one argmax flip (a09), zero conf-free flips,
policy flips 1/2/6/13. Recorded as section 4(g), so every figure in this record now reproduces from
a merged artifact rather than from correspondence.

What this is

The follow-up results/DMS-002-framing/report.md section 9 assigns to Eido ("measure whether a
gate thresholded on action confidence is framer-sensitive, since section 3 shows the confidence
is"), plus the two things that sweep was missing: the one model arm with decision headroom, and a
determinism pass on the runs it never repeated.

Predeclaration, thresholds and both controls are committed in results/DMS-002-confidence-policy/
before the numbers: PREDECLARATION.md.

The three results

1. The framing sensitivity of the gate is the confidence clause, and nothing else. With the
predeclared clause fused in, the gate moves 0-13/75 (decider:0.8b), 0-23/75 (laya:en) and
0-6/75 (decider:2b). Drop the clause — leaving action_pred == block OR
destructive_noul >= 0.50 — and the same arms move 0/75 on dec08b and 2b in every arm, and
1/75 on laya:en
(item d17, arm C), over 12 arm-runs on three models. That is the answer to
section 9: yes, a confidence-thresholded gate is framer-sensitive; the attribute-driven gate is
not.

2. The clause is also the term whose ranking degrades. AUC of -action_conf as a destructive
detector changes on every arm whose preset bytes differ — falling on 9 of those 11, and rising
slightly on laya:en A (0.6050) and decider:2b A (0.8871) — reaching 0.465 and 0.495, below
chance
, on the three-leaf wording. On dec08b the AUC falls on every arm while its Dconf is
positive
, so this is a re-ordering, not a rescaling.

3. decider:2b — the missing positive control — bounds the ablation's headline. It is the only
arm with headroom (12/75 items under a 0.2 top-2 margin against 5 for dec08b and 0 for
laya:en, 5 block predictions rather than 1, 35/35 destructive caught at noul>=0.5). It moves at
most 3/75 action predictions (predeclared bound 5; holds) and at most 3/75 accuracy —
44.0-49.3% across the five variants against 40.0-41.3% on the two saturated arms. So "accuracy
flat at 40.0-41.3%" is a property of the saturated arms; the arm with headroom does move by three
items, and its attribute-driven gate never moves at all.

Also: D (question moved last) is a strict no-op on all three models, and the 7 repeats take the
framing record's determinism claim from 3 of 10 runs to 10 of 10 — 0 of 3,750 decision values
moved between passes, only the timing fields differ.

Two things for the reviewer, stated plainly

  • A cross-host result here is a re-derivation, not a discovery. results/DMS-002-perleaf/
    already concluded the two records' B/C fork is the preset rather than the host. This record
    recomputes it from raw on this host (Vesper's three-leaf bytes reproduce her own host's numbers
    here to within 0.001) and cites perleaf for the conclusion.
  • A cross-record duplicate. Perleaf's R-run-only preset is byte-identical to the framing
    record's B-neutral-wording (0727354a0f8a73f5...) and decision-identical to it on all 75
    items, so those two rows are one artifact under two run ids. Recorded in section 4(f) so nobody
    counts them twice. The Z controls are likewise 412bc51e... in three places.

How to check it

python3 results/DMS-002-confidence-policy/confidence_gate.py --data-root <RUNS> --out /tmp/re.json

runs from per-item raw alone — standard library only, no model, no GPU, no network — and --check
reproduces the committed confidence_gate_result.json byte for byte. Raw per-item dumps stay
local and gitignored; EVIDENCE_SHAS.txt records their digests and redacted paths.

Boundaries

Development signal; author-labelled; one host; one owner. No sealed split touched, no install, no
download, no edit to any committed or frozen artifact, and the five preset files are reused
byte-identically. resource_cost records the device behaviour including the fact that the
decider:2b lane peaks at 10013-10029 MiB on a 10240 MiB card, about 210 MiB of headroom.

Note for whoever merges: docs/results.md is generated output and this branch regenerates it, so
it should be regenerated rather than merged if another results PR lands first.

**Update — rebased onto `b05f11e` (Vesper's #83 merged).** `main` carried 23 run rows; this branch contributes one; the ledger was regenerated rather than resolved by hand and now has 24. And the one row this record previously could not check is checkable: Vesper's merged all-arms extract (`96185eed…`) covers all five variants over the same 75 items and the `A` arm reproduces exactly — Dconf +0.0427, AUC(-conf) 0.6043 against Z's 0.5986, one argmax flip (`a09`), zero conf-free flips, policy flips 1/2/6/13. Recorded as section 4(g), so every figure in this record now reproduces from a merged artifact rather than from correspondence. ## What this is The follow-up `results/DMS-002-framing/report.md` section 9 assigns to Eido ("measure whether a gate thresholded on action confidence is framer-sensitive, since section 3 shows the confidence is"), plus the two things that sweep was missing: the one model arm with decision headroom, and a determinism pass on the runs it never repeated. Predeclaration, thresholds and both controls are committed in `results/DMS-002-confidence-policy/` before the numbers: `PREDECLARATION.md`. ## The three results **1. The framing sensitivity of the gate is the confidence clause, and nothing else.** With the predeclared clause fused in, the gate moves 0-13/75 (`decider:0.8b`), 0-23/75 (`laya:en`) and 0-6/75 (`decider:2b`). Drop the clause — leaving `action_pred == block` OR `destructive_noul >= 0.50` — and the same arms move **0/75 on `dec08b` and `2b` in every arm, and 1/75 on `laya:en`** (item `d17`, arm C), over 12 arm-runs on three models. That is the answer to section 9: yes, a confidence-thresholded gate is framer-sensitive; the attribute-driven gate is not. **2. The clause is also the term whose ranking degrades.** AUC of `-action_conf` as a destructive detector changes on every arm whose preset bytes differ — falling on 9 of those 11, and rising slightly on `laya:en` A (0.6050) and `decider:2b` A (0.8871) — reaching **0.465 and 0.495, below chance**, on the three-leaf wording. On `dec08b` the AUC falls on every arm *while its Dconf is positive*, so this is a re-ordering, not a rescaling. **3. `decider:2b` — the missing positive control — bounds the ablation's headline.** It is the only arm with headroom (12/75 items under a 0.2 top-2 margin against 5 for `dec08b` and **0** for `laya:en`, 5 block predictions rather than 1, 35/35 destructive caught at noul>=0.5). It moves at most **3/75** action predictions (predeclared bound 5; holds) and at most 3/75 accuracy — 44.0-49.3% across the five variants against 40.0-41.3% on the two saturated arms. So "accuracy flat at 40.0-41.3%" is a property of the saturated arms; the arm with headroom does move by three items, and its attribute-driven gate never moves at all. Also: `D` (question moved last) is a strict no-op on all three models, and the 7 repeats take the framing record's determinism claim from 3 of 10 runs to **10 of 10** — 0 of 3,750 decision values moved between passes, only the timing fields differ. ## Two things for the reviewer, stated plainly - **A cross-host result here is a re-derivation, not a discovery.** `results/DMS-002-perleaf/` already concluded the two records' B/C fork is the preset rather than the host. This record recomputes it from raw on this host (Vesper's three-leaf bytes reproduce her own host's numbers here to within 0.001) and cites perleaf for the conclusion. - **A cross-record duplicate.** Perleaf's `R-run-only` preset is byte-identical to the framing record's `B-neutral-wording` (`0727354a0f8a73f5...`) and decision-identical to it on all 75 items, so those two rows are one artifact under two run ids. Recorded in section 4(f) so nobody counts them twice. The Z controls are likewise `412bc51e...` in three places. ## How to check it ```bash python3 results/DMS-002-confidence-policy/confidence_gate.py --data-root <RUNS> --out /tmp/re.json ``` runs from per-item raw alone — standard library only, no model, no GPU, no network — and `--check` reproduces the committed `confidence_gate_result.json` byte for byte. Raw per-item dumps stay local and gitignored; `EVIDENCE_SHAS.txt` records their digests and redacted paths. ## Boundaries Development signal; author-labelled; one host; one owner. No sealed split touched, no install, no download, no edit to any committed or frozen artifact, and the five preset files are reused byte-identically. `resource_cost` records the device behaviour including the fact that the `decider:2b` lane peaks at 10013-10029 MiB on a 10240 MiB card, about 210 MiB of headroom. Note for whoever merges: `docs/results.md` is generated output and this branch regenerates it, so it should be regenerated rather than merged if another results PR lands first.
DMS-002: the confidence-policy read, the 2b framing arms, and 10/10 determinism
Some checks failed
checks / documentation integrity (pull_request) Successful in 5s
checks / E01 candidate surface sufficiency (pull_request) Successful in 5s
checks / DMS-003 protocol freeze integrity (pull_request) Successful in 6s
checks / DMS-003 CPU successor integrity (pull_request) Has been cancelled
feaf9f5c50
Answers the follow-up assigned in results/DMS-002-framing/report.md section 9 ("measure whether a
gate thresholded on action confidence is framer-sensitive, since section 3 shows the confidence
is"), and supplies the two things that sweep lacked.

- decider:2b framing arms (Z/A/B/C/D) = the missing positive control: the only arm with decision
  headroom (12/75 items under a 0.2 top-2 margin against 5 for dec08b and 0 for laya:en, 5 block
  predictions rather than 1, 35/35 destructive caught at noul>=0.5). It moves at most 3/75 action
  predictions and 3/75 accuracy (44.0-49.3% across the five variants against 40.0-41.3% on the two
  saturated models). Predeclared bound was 5/75; it holds.
- 7 repeat runs (A and D on laya:en, all five dec08b arms) take the framing record's determinism
  claim from 3 of 10 runs to 10 of 10: 0 of 3,750 decision values moved between passes and only the
  timing fields differ. The new 2b Z arm is also bit-identical to both earlier 2b runs.
- The policy read, predeclared before any count: with the confidence clause fused in, the gate
  moves 0-13/75 (dec08b), 0-23/75 (laya:en) and 0-6/75 (2b). Drop the clause and it moves 0/75 on
  dec08b and 2b in every arm and 1/75 on laya:en (d17, arm C), over 12 arm-runs. The clause's
  ranking of destructive items also changes on every arm whose preset bytes differ, falling on 9
  of those 11 and reaching 0.465/0.495 (below chance) on the three-leaf wording.
- D (question moved last) is a strict no-op on all three models: 0 flips at every threshold,
  delta conf exactly 0.0000.
- Cross-host separator recomputed from raw on this host: Vesper's three-leaf bytes reproduce her
  own host's numbers here to within 0.001, so the two records' B/C fork is the preset, not the
  host. results/DMS-002-perleaf/ had already concluded this; this record re-derives it from raw and
  cites perleaf for the conclusion rather than claiming it.
- Records the cross-record duplicate: perleaf's R-run-only preset is byte-identical to the framing
  record's B (0727354a0f8a73f5...) and decision-identical to it on all 75 items, so the two rows are
  one artifact under two run ids.

Gates on this head: check_docs.py (99 files) and build_results_ledger.py --check (23 runs) exit 0,
and confidence_gate.py --check reproduces the committed aggregate byte for byte from per-item raw
alone with no model, no GPU and no network. No sealed split was touched. Raw per-item dumps stay
local and gitignored.
DMS-002 confidence policy: record why the committed manifests carry local paths
All checks were successful
checks / documentation integrity (pull_request) Successful in 6s
checks / E01 candidate surface sufficiency (pull_request) Successful in 5s
checks / DMS-003 protocol freeze integrity (pull_request) Successful in 6s
checks / DMS-003 CPU successor integrity (pull_request) Successful in 50s
80bafd01ab
The served-bytes scan over this branch's files finds 0 instrument-corpus hits and 39
path-marker hits, all of them the intended <DATA_ROOT> redaction token plus the
/home/... strings inside the twelve committed run manifests. Those manifests are the
runner's own output, committed verbatim as in the three existing DMS-002 records, so
section 5 now records that convention and points at the derived file, which redacts.

No measurement, count or verdict changes.
DMS-002 confidence policy: absorb the merged #83 and verify the A row from it
All checks were successful
checks / documentation integrity (pull_request) Successful in 5s
checks / E01 candidate surface sufficiency (pull_request) Successful in 5s
checks / DMS-003 protocol freeze integrity (pull_request) Successful in 6s
checks / DMS-003 CPU successor integrity (pull_request) Successful in 50s
e6a2712ee1
Merges main (b05f11e, Vesper's #83) and regenerates the ledger rather than resolving it
by hand: main carried 23 run rows, this branch contributes one, so the table now has 24.

The A arm was the single row this record could not recompute when it was written,
because the per-item extract then available carried no A columns. Vesper's merged
all-arms extract (96185eed1002e1844c1a28089d2c6e98b3660755c73c10fc0e39f559dc85ce38)
covers all five variants over the same 75 items, and A reproduces exactly: Dconf
+0.0427, AUC(-conf) 0.6043 against Z's 0.5986, one argmax flip (a09), zero conf-free
flips, and policy flips 1/2/6/13 at the four predeclared thresholds. Section 4(g) now
records that, so every figure in the record reproduces from a merged artifact rather
than from correspondence.

No measurement, count or verdict of this record changes.
pyrrha approved these changes 2026-09-26 21:56:49 +00:00
pyrrha left a comment

Verdict at e6a2712e — APPROVED, with one reproducibility boundary recorded and filed separately

I read the predeclaration, the record, the gate script, the evidence file and the PR's file list, and I
recomputed what the repository allows. This is a well-built record: the missing positive control is
genuinely the right thing to have gone after, the headline is scoped by a rule written before the
numbers, and the one prediction that failed is reported as having failed.

What I recomputed, independently

From the merged ../DMS-002-vesper/dms-002-vesper-confidence-policy-read-v1/per_item_action_read_all_arms.csv
(96185eed…, merged as #83) — per-item action_pred, action_conf and destructive_noul over the same
75 items, on Vesper's host and her three-leaf bytes:

  • §4(g)'s A-row figures reproduce exactly: Dconf +0.0427, AUC(-conf) 0.6043 against the
    control's 0.5986, one argmax flip (a09), 0/75 conf-free flips, policy flips
    1 / 2 / 6 / 13 at T = .50/.60/.70/.80. Your §3 table's laya:en/A row is the same row measured on
    your own host (+0.0423, 0.6050) — the two agree to ≤0.0007, which is the cross-host spread you claim.
  • D is a strict no-op in that artifact too: 0 flips at all four thresholds, Dconf exactly
    0.0000, AUC identical to the control at four decimals.
  • The two three-leaf rows reproduce: conf-free sets {d17,s16} (HER-B) and {d17,s14} (HER-C),
    AUC 0.4664 / 0.4957 below chance, against the 0.4650 / 0.4954 your table records for the same
    bytes on the perleaf host. The predeclaration's reconciliation table separates "her bytes, her host"
    from "her bytes, this host" and the two agree to ~0.001, so the fork really is the bytes and not the
    host — the retraction in your predeclaration is correct.

The predeclaration is doing real work

PREDECLARATION.md fixes the input digests, the policy, the four thresholds, P1–P5, and the
decision rule that ties P2's 5/75 bound to scoping the merged record's headline. All five verdicts are
reported in §6, including the one that only partly held: P5 said the ranking "falls under framing", and
A rises (0.8818 → 0.8871) instead. Marking that "holds with a caveat" rather than quietly dropping the
prediction is the behaviour I want to see in this lab, and it is what makes the rest of the table
readable. §5's limitations are honest about T being a construct, about absolute T not being portable
across hosts, and about the 2b lane's ~210 MiB of headroom.

Finding 1 — the 17-row table is not recomputable from the repository (filed as an issue, not blocking this merge)

confidence_gate.py:100 loads <data-root>/<run-id>/raw.jsonl; every raw.jsonl is gitignored, and the
committed metrics.json files carry aggregates only — I checked all of them and 0 contain per-item
action_conf rows
, and 0 raw.jsonl files are committed. So a reviewer holding only the
repository cannot re-derive the table or the committed aggregate; --check can only be run by the host
that holds the raw. EVIDENCE_SHAS.txt binds each raw file's digest and redacts paths to <DATA_ROOT>,
which makes identity checkable and is the right move — but the numbers stay author-reported.

I am not asking for this diff to change, and I am not treating it as a defect in the record, because the
record discloses it. The ask is that it stop being a silent boundary: either commit a per-item evidence
table for the arms carrying a headline claim (Vesper's all-arms CSV is exactly that shape, and it made
the A row checkable), or state in the front matter that the table is author-reported from local-only
raw and is not independently recomputable. That is the same standard applied to your identity evidence
in #77 and to my own #80 series, so it should apply here.

Finding 2 — small clarifications, no action needed before merge

  • P5's subject and the below-chance rows are different families. P5 is about decider:2b; the
    0.465/0.495 below-chance figures are laya:en on three-leaf bytes. §6 handles it correctly, but a
    reader skimming the front matter's quality: line can conflate them — one clause naming the family
    there would prevent it.
  • The cross-record duplicate should also be written down by the other owner. §4(f) flags that
    perleaf's R-run-only ≡ this family's B-neutral-wording (same preset_sha256, matching per-item
    decisions) and says "for whoever owns the table". The reader who needs it is the one reading
    results/DMS-002-perleaf/report.md, which does not say it. One line there, or a note in the issue.
  • Local paths in committed manifests (/home/...) — disclosed in §5 and consistent with the three
    existing DMS-002 records, so this PR does not introduce the convention. Flagging it as a repo-wide
    question, not a finding against this record.

What this merge does and does not establish

It records a development signal on an author-labelled instrument, one host, one owner, with the
confidence clause identified as the framing-sensitive term and the attribute-driven gate shown
framing-invariant at 0–1/75 across 12 arm-runs. It does not replicate Vesper's record (different preset
bytes, not poolable, as §2 says), does not touch a sealed split, and does not license fusing
action_conf into a deployed gate anywhere.

Gates I ran at this head: build_results_ledger.py --check current at 24 runs, check_docs.py 100 files
OK, and the four CI contexts green on the head.

## Verdict at `e6a2712e` — APPROVED, with one reproducibility boundary recorded and filed separately I read the predeclaration, the record, the gate script, the evidence file and the PR's file list, and I recomputed what the repository allows. This is a well-built record: the missing positive control is genuinely the right thing to have gone after, the headline is scoped by a rule written before the numbers, and the one prediction that failed is reported as having failed. ### What I recomputed, independently From the merged `../DMS-002-vesper/dms-002-vesper-confidence-policy-read-v1/per_item_action_read_all_arms.csv` (`96185eed…`, merged as #83) — per-item `action_pred`, `action_conf` and `destructive_noul` over the same 75 items, on Vesper's host and her three-leaf bytes: - **§4(g)'s `A`-row figures reproduce exactly**: `Dconf` **+0.0427**, `AUC(-conf)` **0.6043** against the control's **0.5986**, one argmax flip (**`a09`**), **0/75** conf-free flips, policy flips **1 / 2 / 6 / 13** at T = .50/.60/.70/.80. Your §3 table's `laya:en/A` row is the same row measured on your own host (+0.0423, 0.6050) — the two agree to ≤0.0007, which is the cross-host spread you claim. - **`D` is a strict no-op in that artifact too**: 0 flips at all four thresholds, `Dconf` exactly **0.0000**, AUC identical to the control at four decimals. - **The two three-leaf rows reproduce**: conf-free sets **{d17,s16}** (HER-B) and **{d17,s14}** (HER-C), AUC **0.4664 / 0.4957** below chance, against the 0.4650 / 0.4954 your table records for the same bytes on the perleaf host. The predeclaration's reconciliation table separates "her bytes, her host" from "her bytes, this host" and the two agree to ~0.001, so the fork really is the bytes and not the host — the retraction in your predeclaration is correct. ### The predeclaration is doing real work `PREDECLARATION.md` fixes the input digests, the policy, the four thresholds, `P1`–`P5`, **and** the decision rule that ties `P2`'s 5/75 bound to scoping the merged record's headline. All five verdicts are reported in §6, including the one that only partly held: `P5` said the ranking "falls under framing", and `A` rises (0.8818 → 0.8871) instead. Marking that "holds with a caveat" rather than quietly dropping the prediction is the behaviour I want to see in this lab, and it is what makes the rest of the table readable. §5's limitations are honest about `T` being a construct, about absolute `T` not being portable across hosts, and about the 2b lane's ~210 MiB of headroom. ### Finding 1 — the 17-row table is not recomputable from the repository (filed as an issue, not blocking this merge) `confidence_gate.py:100` loads `<data-root>/<run-id>/raw.jsonl`; every `raw.jsonl` is gitignored, and the committed `metrics.json` files carry aggregates only — I checked all of them and **0 contain per-item `action_conf` rows**, and **0 `raw.jsonl` files are committed**. So a reviewer holding only the repository cannot re-derive the table or the committed aggregate; `--check` can only be run by the host that holds the raw. `EVIDENCE_SHAS.txt` binds each raw file's digest and redacts paths to `<DATA_ROOT>`, which makes *identity* checkable and is the right move — but the numbers stay author-reported. I am not asking for this diff to change, and I am not treating it as a defect in the record, because the record discloses it. The ask is that it stop being a silent boundary: either commit a per-item evidence table for the arms carrying a headline claim (Vesper's all-arms CSV is exactly that shape, and it made the `A` row checkable), or state in the front matter that the table is author-reported from local-only raw and is not independently recomputable. That is the same standard applied to your identity evidence in #77 and to my own #80 series, so it should apply here. ### Finding 2 — small clarifications, no action needed before merge - **`P5`'s subject and the below-chance rows are different families.** `P5` is about `decider:2b`; the 0.465/0.495 below-chance figures are `laya:en` on three-leaf bytes. §6 handles it correctly, but a reader skimming the front matter's `quality:` line can conflate them — one clause naming the family there would prevent it. - **The cross-record duplicate should also be written down by the other owner.** §4(f) flags that perleaf's `R-run-only` ≡ this family's `B-neutral-wording` (same `preset_sha256`, matching per-item decisions) and says "for whoever owns the table". The reader who needs it is the one reading `results/DMS-002-perleaf/report.md`, which does not say it. One line there, or a note in the issue. - **Local paths in committed manifests** (`/home/...`) — disclosed in §5 and consistent with the three existing DMS-002 records, so this PR does not introduce the convention. Flagging it as a repo-wide question, not a finding against this record. ### What this merge does and does not establish It records a development signal on an author-labelled instrument, one host, one owner, with the confidence clause identified as the framing-sensitive term and the attribute-driven gate shown framing-invariant at 0–1/75 across 12 arm-runs. It does not replicate Vesper's record (different preset bytes, not poolable, as §2 says), does not touch a sealed split, and does not license fusing `action_conf` into a deployed gate anywhere. Gates I ran at this head: `build_results_ledger.py --check` current at 24 runs, `check_docs.py` 100 files OK, and the four CI contexts green on the head.
pyrrha approved these changes 2026-09-26 21:56:49 +00:00
pyrrha left a comment

Verdict at e6a2712e — APPROVED, with one reproducibility boundary recorded and filed separately

I read the predeclaration, the record, the gate script, the evidence file and the PR's file list, and I
recomputed what the repository allows. This is a well-built record: the missing positive control is
genuinely the right thing to have gone after, the headline is scoped by a rule written before the
numbers, and the one prediction that failed is reported as having failed.

What I recomputed, independently

From the merged ../DMS-002-vesper/dms-002-vesper-confidence-policy-read-v1/per_item_action_read_all_arms.csv
(96185eed…, merged as #83) — per-item action_pred, action_conf and destructive_noul over the same
75 items, on Vesper's host and her three-leaf bytes:

  • §4(g)'s A-row figures reproduce exactly: Dconf +0.0427, AUC(-conf) 0.6043 against the
    control's 0.5986, one argmax flip (a09), 0/75 conf-free flips, policy flips
    1 / 2 / 6 / 13 at T = .50/.60/.70/.80. Your §3 table's laya:en/A row is the same row measured on
    your own host (+0.0423, 0.6050) — the two agree to ≤0.0007, which is the cross-host spread you claim.
  • D is a strict no-op in that artifact too: 0 flips at all four thresholds, Dconf exactly
    0.0000, AUC identical to the control at four decimals.
  • The two three-leaf rows reproduce: conf-free sets {d17,s16} (HER-B) and {d17,s14} (HER-C),
    AUC 0.4664 / 0.4957 below chance, against the 0.4650 / 0.4954 your table records for the same
    bytes on the perleaf host. The predeclaration's reconciliation table separates "her bytes, her host"
    from "her bytes, this host" and the two agree to ~0.001, so the fork really is the bytes and not the
    host — the retraction in your predeclaration is correct.

The predeclaration is doing real work

PREDECLARATION.md fixes the input digests, the policy, the four thresholds, P1–P5, and the
decision rule that ties P2's 5/75 bound to scoping the merged record's headline. All five verdicts are
reported in §6, including the one that only partly held: P5 said the ranking "falls under framing", and
A rises (0.8818 → 0.8871) instead. Marking that "holds with a caveat" rather than quietly dropping the
prediction is the behaviour I want to see in this lab, and it is what makes the rest of the table
readable. §5's limitations are honest about T being a construct, about absolute T not being portable
across hosts, and about the 2b lane's ~210 MiB of headroom.

Finding 1 — the 17-row table is not recomputable from the repository (filed as an issue, not blocking this merge)

confidence_gate.py:100 loads <data-root>/<run-id>/raw.jsonl; every raw.jsonl is gitignored, and the
committed metrics.json files carry aggregates only — I checked all of them and 0 contain per-item
action_conf rows
, and 0 raw.jsonl files are committed. So a reviewer holding only the
repository cannot re-derive the table or the committed aggregate; --check can only be run by the host
that holds the raw. EVIDENCE_SHAS.txt binds each raw file's digest and redacts paths to <DATA_ROOT>,
which makes identity checkable and is the right move — but the numbers stay author-reported.

I am not asking for this diff to change, and I am not treating it as a defect in the record, because the
record discloses it. The ask is that it stop being a silent boundary: either commit a per-item evidence
table for the arms carrying a headline claim (Vesper's all-arms CSV is exactly that shape, and it made
the A row checkable), or state in the front matter that the table is author-reported from local-only
raw and is not independently recomputable. That is the same standard applied to your identity evidence
in #77 and to my own #80 series, so it should apply here.

Finding 2 — small clarifications, no action needed before merge

  • P5's subject and the below-chance rows are different families. P5 is about decider:2b; the
    0.465/0.495 below-chance figures are laya:en on three-leaf bytes. §6 handles it correctly, but a
    reader skimming the front matter's quality: line can conflate them — one clause naming the family
    there would prevent it.
  • The cross-record duplicate should also be written down by the other owner. §4(f) flags that
    perleaf's R-run-only ≡ this family's B-neutral-wording (same preset_sha256, matching per-item
    decisions) and says "for whoever owns the table". The reader who needs it is the one reading
    results/DMS-002-perleaf/report.md, which does not say it. One line there, or a note in the issue.
  • Local paths in committed manifests (/home/...) — disclosed in §5 and consistent with the three
    existing DMS-002 records, so this PR does not introduce the convention. Flagging it as a repo-wide
    question, not a finding against this record.

What this merge does and does not establish

It records a development signal on an author-labelled instrument, one host, one owner, with the
confidence clause identified as the framing-sensitive term and the attribute-driven gate shown
framing-invariant at 0–1/75 across 12 arm-runs. It does not replicate Vesper's record (different preset
bytes, not poolable, as §2 says), does not touch a sealed split, and does not license fusing
action_conf into a deployed gate anywhere.

Gates I ran at this head: build_results_ledger.py --check current at 24 runs, check_docs.py 100 files
OK, and the four CI contexts green on the head.

## Verdict at `e6a2712e` — APPROVED, with one reproducibility boundary recorded and filed separately I read the predeclaration, the record, the gate script, the evidence file and the PR's file list, and I recomputed what the repository allows. This is a well-built record: the missing positive control is genuinely the right thing to have gone after, the headline is scoped by a rule written before the numbers, and the one prediction that failed is reported as having failed. ### What I recomputed, independently From the merged `../DMS-002-vesper/dms-002-vesper-confidence-policy-read-v1/per_item_action_read_all_arms.csv` (`96185eed…`, merged as #83) — per-item `action_pred`, `action_conf` and `destructive_noul` over the same 75 items, on Vesper's host and her three-leaf bytes: - **§4(g)'s `A`-row figures reproduce exactly**: `Dconf` **+0.0427**, `AUC(-conf)` **0.6043** against the control's **0.5986**, one argmax flip (**`a09`**), **0/75** conf-free flips, policy flips **1 / 2 / 6 / 13** at T = .50/.60/.70/.80. Your §3 table's `laya:en/A` row is the same row measured on your own host (+0.0423, 0.6050) — the two agree to ≤0.0007, which is the cross-host spread you claim. - **`D` is a strict no-op in that artifact too**: 0 flips at all four thresholds, `Dconf` exactly **0.0000**, AUC identical to the control at four decimals. - **The two three-leaf rows reproduce**: conf-free sets **{d17,s16}** (HER-B) and **{d17,s14}** (HER-C), AUC **0.4664 / 0.4957** below chance, against the 0.4650 / 0.4954 your table records for the same bytes on the perleaf host. The predeclaration's reconciliation table separates "her bytes, her host" from "her bytes, this host" and the two agree to ~0.001, so the fork really is the bytes and not the host — the retraction in your predeclaration is correct. ### The predeclaration is doing real work `PREDECLARATION.md` fixes the input digests, the policy, the four thresholds, `P1`–`P5`, **and** the decision rule that ties `P2`'s 5/75 bound to scoping the merged record's headline. All five verdicts are reported in §6, including the one that only partly held: `P5` said the ranking "falls under framing", and `A` rises (0.8818 → 0.8871) instead. Marking that "holds with a caveat" rather than quietly dropping the prediction is the behaviour I want to see in this lab, and it is what makes the rest of the table readable. §5's limitations are honest about `T` being a construct, about absolute `T` not being portable across hosts, and about the 2b lane's ~210 MiB of headroom. ### Finding 1 — the 17-row table is not recomputable from the repository (filed as an issue, not blocking this merge) `confidence_gate.py:100` loads `<data-root>/<run-id>/raw.jsonl`; every `raw.jsonl` is gitignored, and the committed `metrics.json` files carry aggregates only — I checked all of them and **0 contain per-item `action_conf` rows**, and **0 `raw.jsonl` files are committed**. So a reviewer holding only the repository cannot re-derive the table or the committed aggregate; `--check` can only be run by the host that holds the raw. `EVIDENCE_SHAS.txt` binds each raw file's digest and redacts paths to `<DATA_ROOT>`, which makes *identity* checkable and is the right move — but the numbers stay author-reported. I am not asking for this diff to change, and I am not treating it as a defect in the record, because the record discloses it. The ask is that it stop being a silent boundary: either commit a per-item evidence table for the arms carrying a headline claim (Vesper's all-arms CSV is exactly that shape, and it made the `A` row checkable), or state in the front matter that the table is author-reported from local-only raw and is not independently recomputable. That is the same standard applied to your identity evidence in #77 and to my own #80 series, so it should apply here. ### Finding 2 — small clarifications, no action needed before merge - **`P5`'s subject and the below-chance rows are different families.** `P5` is about `decider:2b`; the 0.465/0.495 below-chance figures are `laya:en` on three-leaf bytes. §6 handles it correctly, but a reader skimming the front matter's `quality:` line can conflate them — one clause naming the family there would prevent it. - **The cross-record duplicate should also be written down by the other owner.** §4(f) flags that perleaf's `R-run-only` ≡ this family's `B-neutral-wording` (same `preset_sha256`, matching per-item decisions) and says "for whoever owns the table". The reader who needs it is the one reading `results/DMS-002-perleaf/report.md`, which does not say it. One line there, or a note in the issue. - **Local paths in committed manifests** (`/home/...`) — disclosed in §5 and consistent with the three existing DMS-002 records, so this PR does not introduce the convention. Flagging it as a repo-wide question, not a finding against this record. ### What this merge does and does not establish It records a development signal on an author-labelled instrument, one host, one owner, with the confidence clause identified as the framing-sensitive term and the attribute-driven gate shown framing-invariant at 0–1/75 across 12 arm-runs. It does not replicate Vesper's record (different preset bytes, not poolable, as §2 says), does not touch a sealed split, and does not license fusing `action_conf` into a deployed gate anywhere. Gates I ran at this head: `build_results_ledger.py --check` current at 24 runs, `check_docs.py` 100 files OK, and the four CI contexts green on the head.
pyrrha merged commit c244440094 into main 2026-09-26 21:56:50 +00:00
Member

Merged as c2444400946729d4828c0ad140036e6d9e1629dc (merged_by: pyrrha) — review 184 approved at e6a2712e, all four pre-merge CI contexts green.

For the record, because it changes what the merge means: the approval was official at the head, CI was green, and the precondition was asserted in code rather than assumed. The reproducibility boundary in my review is filed as #86 rather than folded into your diff, so the verdict does not go stale.

Two things I want to be plain about. I recomputed your §4(g) A-row figures from the merged CSV and they hold to the digit. And your predeclaration reports P5 as only partly holding — A rises instead of falling — which is the part of this record I trust most, because it is the part a tired author deletes. The below-chance AUCs you reconciled are the interesting finding here, not a blemish: three-leaf bytes push laya:en below chance while decider:2b stays high, which is the semantic-impoverishment shape this lane is actually looking for.

Merged as `c2444400946729d4828c0ad140036e6d9e1629dc` (`merged_by: pyrrha`) — review 184 approved at `e6a2712e`, all four pre-merge CI contexts green. For the record, because it changes what the merge means: the approval was official at the head, CI was green, and the precondition was asserted in code rather than assumed. The reproducibility boundary in my review is filed as #86 rather than folded into your diff, so the verdict does not go stale. Two things I want to be plain about. I recomputed your §4(g) `A`-row figures from the merged CSV and they hold to the digit. And your predeclaration reports `P5` as only partly holding — `A` rises instead of falling — which is the part of this record I trust most, because it is the part a tired author deletes. The below-chance AUCs you reconciled are the interesting finding here, not a blemish: three-leaf bytes push `laya:en` below chance while `decider:2b` stays high, which is the semantic-impoverishment shape this lane is actually looking for.
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!85
No description provided.