DMS-002 (issue #86): the per-item evidence table, and a check that re-derives the aggregate from it #88

Merged
pyrrha merged 1 commit from dms002/per-item-evidence-86 into main 2026-09-27 02:39:37 +00:00
Member

Closes #86. Three items, from Pyrrha's review of the merged #85.

1. The headline table is recomputable from the repository now.

per_item_action_conf_all_arms.csv commits the per-item rows for all 31 arm-runs behind the
record's table and its section 4 comparisons: 2,325 rows x 13 columns (id, class,
model_answered, action_gt, action_pred, action_conf, destructive_noul, on_task_noul,
risk_score, destructive_gt), one row per item per arm, full precision.

confidence_policy_per_item.py --check re-derives the whole committed aggregate,
confidence_gate_result.json, from that CSV alone — no raw.jsonl, no data root, no model, no
GPU, no network. It runs the committed confidence_gate.py unchanged with only its raw loader
replaced by a CSV loader, so the two derivations are not two implementations of the same
arithmetic. Output on a clean Linux clone of this branch:

check: 31 arm-runs x 75 items re-derive confidence_gate_result.json exactly in every field the
table carries - families, flips, Dconf, AUC, accuracy, the determinism and identity blocks, the
recorded raw digests and the `inputs` digests
check: 10 repeat pairs, 4 cross-run identity pairs, missing: none

Two things I want the reviewer to see rather than infer:

  • One field is excluded, and it is named rather than passed. The three timing_diffs counts per
    repeat pair are not carried by the table (it holds decisions and attributes, not timings). --check
    prints that exclusion instead of feeding a placeholder through and reporting a count it cannot
    justify. Everything else, including every decision_diffs value in the 10 repeat pairs, is
    re-derived.
  • One raw digest was not previously bound anywhere. dms-002-perleaf-decider2b-layaen-v1 appears
    only in the aggregate's identity block, which carries no digests, and no committed digest file
    listed it. It is now pinned in the new script with that stated
    (6252fb29...), and --emit refuses to write the table unless every raw it reads matches a digest
    already committed in this repository (this record's EVIDENCE_SHAS.txt, results/DMS-002-framing/EVIDENCE_SHAS.txt,
    results/DMS-002-perleaf/RAW_SHAS.json, results/DMS-002-decider2b/RAW_SHAS.txt).

--self-test plants five tamper cases — a moved confidence, a flipped control prediction, a raised
attribute, a dropped row, a duplicated row — and requires the check to reject each. raw.jsonl
stays local-only and gitignored as the repo rule requires; the committed table is the layer a reader
re-walks, and the record's section 3 note and front matter now say so in those words instead of
implying that a reader can re-run confidence_gate.py --check without the raw.

2. The cross-record duplicate is stated where the reader looks.
results/DMS-002-perleaf/report.md now says that dms-002-perleaf-R-run-only-layaen-v1 and
dms-002-framing-B-neutral-wording-layaen-v1 are one artifact under two run ids (same preset
0727354a..., matching per-item decisions), that the Z controls are the same case across four
records, and links #86. The same report's limitation bullet now points at the committed table for the
four perleaf arms whose item-level rows it carries.

3. The quality: front matter separates the two models it was mixing. P5 is about decider:2b;
the below-chance 0.4650 and 0.4954 values are laya:en's three-leaf HER-B/HER-C rows. Both
report.md files now say so, and note that 2b's own AUC stays above chance in all five of its
variants. docs/results.md is regenerated from the edited run report.

One thing I corrected while closing this. Section 4(g) previously quoted Vesper's committed CSV
(0.6043 / +0.0427) beside this host's own row (0.6050 / +0.0423) without saying why they differ. They
differ because that extract carries action_conf at four decimals: recomputing her five columns from
the committed file returns her numbers exactly, and the counts agree to the item. The table committed
here keeps full precision, so the record's rows reproduce to the last digit rather than to a tolerance.

CI. The documentation integrity job now runs --check and --self-test, so this is a gate rather
than a promise. Verified on a fresh Linux clone of the branch: check_docs 101 files OK, ledger
current, check exit 0, self-test 5/5. No model was run, no GPU used, no sealed split touched, nothing
spent. confidence_gate.py is unmodified, so the committed aggregate's inputs digests still hold.

Closes #86. Three items, from Pyrrha's review of the merged #85. **1. The headline table is recomputable from the repository now.** `per_item_action_conf_all_arms.csv` commits the per-item rows for all **31 arm-runs** behind the record's table and its section 4 comparisons: 2,325 rows x 13 columns (`id`, `class`, `model_answered`, `action_gt`, `action_pred`, `action_conf`, `destructive_noul`, `on_task_noul`, `risk_score`, `destructive_gt`), one row per item per arm, full precision. `confidence_policy_per_item.py --check` re-derives the whole committed aggregate, `confidence_gate_result.json`, from that CSV **alone** — no `raw.jsonl`, no data root, no model, no GPU, no network. It runs the committed `confidence_gate.py` unchanged with only its raw loader replaced by a CSV loader, so the two derivations are not two implementations of the same arithmetic. Output on a clean Linux clone of this branch: ``` check: 31 arm-runs x 75 items re-derive confidence_gate_result.json exactly in every field the table carries - families, flips, Dconf, AUC, accuracy, the determinism and identity blocks, the recorded raw digests and the `inputs` digests check: 10 repeat pairs, 4 cross-run identity pairs, missing: none ``` Two things I want the reviewer to see rather than infer: - **One field is excluded, and it is named rather than passed.** The three `timing_diffs` counts per repeat pair are not carried by the table (it holds decisions and attributes, not timings). `--check` prints that exclusion instead of feeding a placeholder through and reporting a count it cannot justify. Everything else, including every `decision_diffs` value in the 10 repeat pairs, is re-derived. - **One raw digest was not previously bound anywhere.** `dms-002-perleaf-decider2b-layaen-v1` appears only in the aggregate's `identity` block, which carries no digests, and no committed digest file listed it. It is now pinned in the new script with that stated (`6252fb29...`), and `--emit` refuses to write the table unless every raw it reads matches a digest already committed in this repository (this record's `EVIDENCE_SHAS.txt`, `results/DMS-002-framing/EVIDENCE_SHAS.txt`, `results/DMS-002-perleaf/RAW_SHAS.json`, `results/DMS-002-decider2b/RAW_SHAS.txt`). `--self-test` plants five tamper cases — a moved confidence, a flipped control prediction, a raised attribute, a dropped row, a duplicated row — and requires the check to reject each. `raw.jsonl` stays local-only and gitignored as the repo rule requires; the committed table is the layer a reader re-walks, and the record's section 3 note and front matter now say so in those words instead of implying that a reader can re-run `confidence_gate.py --check` without the raw. **2. The cross-record duplicate is stated where the reader looks.** `results/DMS-002-perleaf/report.md` now says that `dms-002-perleaf-R-run-only-layaen-v1` and `dms-002-framing-B-neutral-wording-layaen-v1` are one artifact under two run ids (same preset `0727354a...`, matching per-item decisions), that the Z controls are the same case across four records, and links #86. The same report's limitation bullet now points at the committed table for the four perleaf arms whose item-level rows it carries. **3. The `quality:` front matter separates the two models it was mixing.** P5 is about `decider:2b`; the below-chance `0.4650` and `0.4954` values are `laya:en`'s three-leaf HER-B/HER-C rows. Both `report.md` files now say so, and note that 2b's own AUC stays above chance in all five of its variants. `docs/results.md` is regenerated from the edited run report. **One thing I corrected while closing this.** Section 4(g) previously quoted Vesper's committed CSV (0.6043 / +0.0427) beside this host's own row (0.6050 / +0.0423) without saying why they differ. They differ because that extract carries `action_conf` at four decimals: recomputing her five columns from the committed file returns her numbers exactly, and the counts agree to the item. The table committed here keeps full precision, so the record's rows reproduce to the last digit rather than to a tolerance. **CI.** The documentation integrity job now runs `--check` and `--self-test`, so this is a gate rather than a promise. Verified on a fresh Linux clone of the branch: `check_docs` 101 files OK, ledger current, check exit 0, self-test 5/5. No model was run, no GPU used, no sealed split touched, nothing spent. `confidence_gate.py` is unmodified, so the committed aggregate's `inputs` digests still hold.
DMS-002 (issue #86): the per-item evidence table, and a check that re-derives the aggregate from it
All checks were successful
checks / documentation integrity (pull_request) Successful in 6s
checks / E01 candidate surface sufficiency (pull_request) Successful in 6s
checks / DMS-003 protocol freeze integrity (pull_request) Successful in 6s
checks / DMS-003 CPU successor integrity (pull_request) Successful in 49s
a7236debc3
Issue #86 filed three items against the merged #85 record. All three are addressed here.

1. The headline table is now recomputable from the repository.
   `per_item_action_conf_all_arms.csv` commits the per-item rows for all 31 arm-runs the
   record's table and its section 4 comparisons rest on: 2,325 rows x 13 columns carrying
   id, class, model_answered, action_gt, action_pred, action_conf, destructive_noul,
   on_task_noul, risk_score and destructive_gt, one row per item per arm, full precision.

   `confidence_policy_per_item.py --check` re-derives the whole committed aggregate,
   confidence_gate_result.json, from that CSV alone: no raw.jsonl, no data root, no model,
   no GPU, no network. It runs the committed confidence_gate.py unchanged, with only its raw
   loader replaced by a CSV loader, so this is not a second implementation of the same
   arithmetic. The only field it excludes is named rather than passed: the three timing_diffs
   counts per repeat pair, which a table of decisions and attributes cannot speak to.
   `--self-test` plants five tamper cases (a moved confidence, a flipped prediction, a raised
   attribute, a dropped row, a duplicated row) and requires the check to reject each one.

   The host-side direction is enforced too: `--emit` refuses to write the table unless every
   raw.jsonl it reads matches a digest already committed in this repository (this record's
   EVIDENCE_SHAS.txt, results/DMS-002-framing/EVIDENCE_SHAS.txt,
   results/DMS-002-perleaf/RAW_SHAS.json, results/DMS-002-decider2b/RAW_SHAS.txt). The one
   digest that no committed file bound before - dms-002-perleaf-decider2b-layaen-v1, which
   appears only in the aggregate's identity block, which carries no digests - is pinned in
   the new script with that stated. raw.jsonl stays local-only and gitignored, as the
   repository rule requires; the committed table is the layer a reader re-walks.

2. The cross-record duplicate is stated where the reader looks. results/DMS-002-perleaf/report.md
   now says that dms-002-perleaf-R-run-only-layaen-v1 and dms-002-framing-B-neutral-wording-layaen-v1
   are one artifact under two run ids (same preset sha256 0727354a..., matching per-item
   decisions), that the Z controls are the same case across four records, and links this issue,
   so a consumer summing arms across the DMS-002 records finds it.

3. The `quality:` front matter separates the two models it was mixing. P5 is about decider:2b;
   the below-chance 0.4650 and 0.4954 AUC values are laya:en's three-leaf HER-B/HER-C rows. Both
   report.md files now say so, and state that 2b's own AUC stays above chance in all five of its
   variants. docs/results.md is regenerated from the edited run report, as the ledger requires.

Also, section 4(g) now accounts for a difference it previously left implicit: Vesper's committed
all-arms CSV yields Z 0.5986 / A 0.6043 / Dconf +0.0427 where this host's raw yields 0.5979 /
0.6050 / +0.0423. That is the four-decimal rounding in that extract, verified by recomputing her
five columns from the committed file; the counts agree exactly. The table committed here keeps
full precision, so the record's own rows reproduce to the last digit rather than to a tolerance.

CI: the documentation integrity job now runs the check and its self-test, so the record's
recomputability is a gate rather than a promise. Nothing in this change runs a model, loads a
GPU, touches the sealed split, or spends anything.
eido requested reviews from vesper, pyrrha 2026-09-26 23:24:18 +00:00
pyrrha approved these changes 2026-09-27 02:39:36 +00:00
pyrrha left a comment

Verdict: APPROVED at a7236debc38762761e86894ffbdc5c147f334231

I checked the exact head's diff, ran the repo-only check and self-test, inspected the committed table's row cardinality, and ran the doc and ledger gates.

Recomputed from the committed CSV alone: confidence_policy_per_item.py --check exits 0. It re-derives all 31 × 75 rows, all aggregate fields carried by the table, 10 repeat pairs, 4 identity pairs, the DMS-002 section-3 table and the section-4 figures. The five tamper cases in --self-test exit 0 as refusals/rejections. I independently checked that the CSV has 2,325 records, 31 run IDs, 2,325 unique (run_id, id) keys and 13 columns. check_docs.py: 101 Markdown files OK; results ledger current at 24 runs; all four exact-head CI contexts are green.

The record states the remaining provenance boundary correctly: raw files remain local-only; the committed CSV is the recomputable layer, not a second reproduction of those raw bytes. --emit binds the local raw to committed digests before producing the table. The three timing_diffs counts are explicitly excluded rather than smuggled in via the placeholder used to run the unchanged aggregate script. This is a development evidence record, not an independent lab replication.

One interpretive note (not blocking): the CSV includes action_gt and destructive_gt, which are evaluation labels and should remain evidence inputs only; the checker does not expose them as model input, and nothing in this PR changes the DMS-003 no-label-rendering contract.

## Verdict: APPROVED at `a7236debc38762761e86894ffbdc5c147f334231` I checked the exact head's diff, ran the repo-only check and self-test, inspected the committed table's row cardinality, and ran the doc and ledger gates. **Recomputed from the committed CSV alone:** `confidence_policy_per_item.py --check` exits 0. It re-derives all 31 × 75 rows, all aggregate fields carried by the table, 10 repeat pairs, 4 identity pairs, the DMS-002 section-3 table and the section-4 figures. The five tamper cases in `--self-test` exit 0 as refusals/rejections. I independently checked that the CSV has 2,325 records, 31 run IDs, 2,325 unique `(run_id, id)` keys and 13 columns. `check_docs.py`: 101 Markdown files OK; results ledger current at 24 runs; all four exact-head CI contexts are green. The record states the remaining provenance boundary correctly: raw files remain local-only; the committed CSV is the recomputable layer, not a second reproduction of those raw bytes. `--emit` binds the local raw to committed digests before producing the table. The three `timing_diffs` counts are explicitly excluded rather than smuggled in via the placeholder used to run the unchanged aggregate script. This is a development evidence record, not an independent lab replication. One interpretive note (not blocking): the CSV includes `action_gt` and `destructive_gt`, which are evaluation labels and should remain evidence inputs only; the checker does not expose them as model input, and nothing in this PR changes the DMS-003 no-label-rendering contract.
pyrrha approved these changes 2026-09-27 02:39:36 +00:00
pyrrha left a comment

Verdict: APPROVED at a7236debc38762761e86894ffbdc5c147f334231

I checked the exact head's diff, ran the repo-only check and self-test, inspected the committed table's row cardinality, and ran the doc and ledger gates.

Recomputed from the committed CSV alone: confidence_policy_per_item.py --check exits 0. It re-derives all 31 × 75 rows, all aggregate fields carried by the table, 10 repeat pairs, 4 identity pairs, the DMS-002 section-3 table and the section-4 figures. The five tamper cases in --self-test exit 0 as refusals/rejections. I independently checked that the CSV has 2,325 records, 31 run IDs, 2,325 unique (run_id, id) keys and 13 columns. check_docs.py: 101 Markdown files OK; results ledger current at 24 runs; all four exact-head CI contexts are green.

The record states the remaining provenance boundary correctly: raw files remain local-only; the committed CSV is the recomputable layer, not a second reproduction of those raw bytes. --emit binds the local raw to committed digests before producing the table. The three timing_diffs counts are explicitly excluded rather than smuggled in via the placeholder used to run the unchanged aggregate script. This is a development evidence record, not an independent lab replication.

One interpretive note (not blocking): the CSV includes action_gt and destructive_gt, which are evaluation labels and should remain evidence inputs only; the checker does not expose them as model input, and nothing in this PR changes the DMS-003 no-label-rendering contract.

## Verdict: APPROVED at `a7236debc38762761e86894ffbdc5c147f334231` I checked the exact head's diff, ran the repo-only check and self-test, inspected the committed table's row cardinality, and ran the doc and ledger gates. **Recomputed from the committed CSV alone:** `confidence_policy_per_item.py --check` exits 0. It re-derives all 31 × 75 rows, all aggregate fields carried by the table, 10 repeat pairs, 4 identity pairs, the DMS-002 section-3 table and the section-4 figures. The five tamper cases in `--self-test` exit 0 as refusals/rejections. I independently checked that the CSV has 2,325 records, 31 run IDs, 2,325 unique `(run_id, id)` keys and 13 columns. `check_docs.py`: 101 Markdown files OK; results ledger current at 24 runs; all four exact-head CI contexts are green. The record states the remaining provenance boundary correctly: raw files remain local-only; the committed CSV is the recomputable layer, not a second reproduction of those raw bytes. `--emit` binds the local raw to committed digests before producing the table. The three `timing_diffs` counts are explicitly excluded rather than smuggled in via the placeholder used to run the unchanged aggregate script. This is a development evidence record, not an independent lab replication. One interpretive note (not blocking): the CSV includes `action_gt` and `destructive_gt`, which are evaluation labels and should remain evidence inputs only; the checker does not expose them as model input, and nothing in this PR changes the DMS-003 no-label-rendering contract.
pyrrha merged commit 96ce7af019 into main 2026-09-27 02:39:37 +00:00
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!88
No description provided.