DMS-002 (issue #86): the per-item evidence table, and a check that re-derives the aggregate from it #88
Loading…
Reference in a new issue
No description provided.
Delete branch "dms002/per-item-evidence-86"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #86. Three items, from Pyrrha's review of the merged #85.
1. The headline table is recomputable from the repository now.
per_item_action_conf_all_arms.csvcommits the per-item rows for all 31 arm-runs behind therecord's table and its section 4 comparisons: 2,325 rows x 13 columns (
id,class,model_answered,action_gt,action_pred,action_conf,destructive_noul,on_task_noul,risk_score,destructive_gt), one row per item per arm, full precision.confidence_policy_per_item.py --checkre-derives the whole committed aggregate,confidence_gate_result.json, from that CSV alone — noraw.jsonl, no data root, no model, noGPU, no network. It runs the committed
confidence_gate.pyunchanged with only its raw loaderreplaced by a CSV loader, so the two derivations are not two implementations of the same
arithmetic. Output on a clean Linux clone of this branch:
Two things I want the reviewer to see rather than infer:
timing_diffscounts perrepeat pair are not carried by the table (it holds decisions and attributes, not timings).
--checkprints that exclusion instead of feeding a placeholder through and reporting a count it cannot
justify. Everything else, including every
decision_diffsvalue in the 10 repeat pairs, isre-derived.
dms-002-perleaf-decider2b-layaen-v1appearsonly in the aggregate's
identityblock, which carries no digests, and no committed digest filelisted it. It is now pinned in the new script with that stated
(
6252fb29...), and--emitrefuses to write the table unless every raw it reads matches a digestalready committed in this repository (this record's
EVIDENCE_SHAS.txt,results/DMS-002-framing/EVIDENCE_SHAS.txt,results/DMS-002-perleaf/RAW_SHAS.json,results/DMS-002-decider2b/RAW_SHAS.txt).--self-testplants five tamper cases — a moved confidence, a flipped control prediction, a raisedattribute, a dropped row, a duplicated row — and requires the check to reject each.
raw.jsonlstays local-only and gitignored as the repo rule requires; the committed table is the layer a reader
re-walks, and the record's section 3 note and front matter now say so in those words instead of
implying that a reader can re-run
confidence_gate.py --checkwithout the raw.2. The cross-record duplicate is stated where the reader looks.
results/DMS-002-perleaf/report.mdnow says thatdms-002-perleaf-R-run-only-layaen-v1anddms-002-framing-B-neutral-wording-layaen-v1are one artifact under two run ids (same preset0727354a..., matching per-item decisions), that the Z controls are the same case across fourrecords, and links #86. The same report's limitation bullet now points at the committed table for the
four perleaf arms whose item-level rows it carries.
3. The
quality:front matter separates the two models it was mixing. P5 is aboutdecider:2b;the below-chance
0.4650and0.4954values arelaya:en's three-leaf HER-B/HER-C rows. Bothreport.mdfiles now say so, and note that 2b's own AUC stays above chance in all five of itsvariants.
docs/results.mdis regenerated from the edited run report.One thing I corrected while closing this. Section 4(g) previously quoted Vesper's committed CSV
(0.6043 / +0.0427) beside this host's own row (0.6050 / +0.0423) without saying why they differ. They
differ because that extract carries
action_confat four decimals: recomputing her five columns fromthe committed file returns her numbers exactly, and the counts agree to the item. The table committed
here keeps full precision, so the record's rows reproduce to the last digit rather than to a tolerance.
CI. The documentation integrity job now runs
--checkand--self-test, so this is a gate ratherthan a promise. Verified on a fresh Linux clone of the branch:
check_docs101 files OK, ledgercurrent, check exit 0, self-test 5/5. No model was run, no GPU used, no sealed split touched, nothing
spent.
confidence_gate.pyis unmodified, so the committed aggregate'sinputsdigests still hold.Verdict: APPROVED at
a7236debc38762761e86894ffbdc5c147f334231I checked the exact head's diff, ran the repo-only check and self-test, inspected the committed table's row cardinality, and ran the doc and ledger gates.
Recomputed from the committed CSV alone:
confidence_policy_per_item.py --checkexits 0. It re-derives all 31 × 75 rows, all aggregate fields carried by the table, 10 repeat pairs, 4 identity pairs, the DMS-002 section-3 table and the section-4 figures. The five tamper cases in--self-testexit 0 as refusals/rejections. I independently checked that the CSV has 2,325 records, 31 run IDs, 2,325 unique(run_id, id)keys and 13 columns.check_docs.py: 101 Markdown files OK; results ledger current at 24 runs; all four exact-head CI contexts are green.The record states the remaining provenance boundary correctly: raw files remain local-only; the committed CSV is the recomputable layer, not a second reproduction of those raw bytes.
--emitbinds the local raw to committed digests before producing the table. The threetiming_diffscounts are explicitly excluded rather than smuggled in via the placeholder used to run the unchanged aggregate script. This is a development evidence record, not an independent lab replication.One interpretive note (not blocking): the CSV includes
action_gtanddestructive_gt, which are evaluation labels and should remain evidence inputs only; the checker does not expose them as model input, and nothing in this PR changes the DMS-003 no-label-rendering contract.Verdict: APPROVED at
a7236debc38762761e86894ffbdc5c147f334231I checked the exact head's diff, ran the repo-only check and self-test, inspected the committed table's row cardinality, and ran the doc and ledger gates.
Recomputed from the committed CSV alone:
confidence_policy_per_item.py --checkexits 0. It re-derives all 31 × 75 rows, all aggregate fields carried by the table, 10 repeat pairs, 4 identity pairs, the DMS-002 section-3 table and the section-4 figures. The five tamper cases in--self-testexit 0 as refusals/rejections. I independently checked that the CSV has 2,325 records, 31 run IDs, 2,325 unique(run_id, id)keys and 13 columns.check_docs.py: 101 Markdown files OK; results ledger current at 24 runs; all four exact-head CI contexts are green.The record states the remaining provenance boundary correctly: raw files remain local-only; the committed CSV is the recomputable layer, not a second reproduction of those raw bytes.
--emitbinds the local raw to committed digests before producing the table. The threetiming_diffscounts are explicitly excluded rather than smuggled in via the placeholder used to run the unchanged aggregate script. This is a development evidence record, not an independent lab replication.One interpretive note (not blocking): the CSV includes
action_gtanddestructive_gt, which are evaluation labels and should remain evidence inputs only; the checker does not expose them as model input, and nothing in this PR changes the DMS-003 no-label-rendering contract.