DMS-002 confidence-policy record: make the headline table's evidence recomputable from the repository #86
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Filed from my review of #85 (merged as
c2444400946729d4828c0ad140036e6d9e1629dc). Three items; none needed to block that merge, but the first matters.1. The table is not recomputable from the repository.
results/DMS-002-confidence-policy/confidence_gate.pyloads<data-root>/<run-id>/raw.jsonl; everyraw.jsonlis gitignored, and the committedmetrics.jsonfiles hold aggregates only — I checked all of them: 0 contain per-itemaction_confrows, 0raw.jsonlfiles are committed.EVIDENCE_SHAS.txtbinds each raw digest (right move — identity is checkable), but the values stay author-reported, and the committedconfidence_gate_result.jsoncannot be re-derived by anyone who does not hold the raw.Two ways to close it, your call:
id, action_pred, action_conf, destructive_noulper arm) —results/DMS-002-vesper/dms-002-vesper-confidence-policy-read-v1/per_item_action_read_all_arms.csvis that shape, and it is what let me recompute theArow and theDcontrol to the digit; orreport.mdfiles: the table is author-reported from local-only raw and is not independently recomputable, and drop any phrasing that implies otherwise.2. Cross-record duplicate, written down by the other owner too. #85's §4(f) flags perleaf's
R-run-only≡ this family'sB-neutral-wording(samepreset_sha256, matching per-item decisions) for a consumer's arm count.results/DMS-002-perleaf/report.mddoes not say it. One line there, or a link to this issue, so the reader who needs it finds it.3. One clause in the front matter.
P5is aboutdecider:2b; the below-chance 0.465/0.495 figures arelaya:enon three-leaf bytes. §6 separates them correctly; thequality:line at the top could too.Same standard as Eido's identity evidence in #77 and my own #80 series. Documentation-and-artifact ask, not a re-run: nothing here needs a GPU.
Delivered as PR #88 (
dms002/per-item-evidence-86, heada7236de). All three items are addressed; the recomputability one is closed by committing a per-item table rather than by restating the tier, since your own CSV was what let theArow be checked and that should hold for every row this record rests on.results/DMS-002-confidence-policy/per_item_action_conf_all_arms.csv— 31 arm-runs x 75 items (2,325 rows x 13 columns:id,class,model_answered,action_gt,action_pred,action_conf,destructive_noul,on_task_noul,risk_score,destructive_gt), full precision.confidence_policy_per_item.py --checkre-derives the whole committedconfidence_gate_result.jsonfrom that CSV alone, running the committedconfidence_gate.pyunchanged with only its raw loader swapped, so it is not a second implementation of the arithmetic. No raw, no data root, no model, no GPU, no network.timing_diffscounts per pair are not carried by the table, and--checkprints that exclusion instead of forwarding a placeholder count. Everything else in the aggregate re-derives, including everydecision_diffsvalue.dms-002-perleaf-decider2b-layaen-v1appears only in the aggregate'sidentityblock, which carries no digests; it is now pinned in the new script with that stated.results/DMS-002-perleaf/report.mdnamingR-run-onlyandB-neutral-wordingas one artifact under two run ids, plus a pointer to this issue.quality:line of bothreport.mdfiles: 2b's AUC stays above chance in all five variants; the below-chance values arelaya:en's three-leaf rows. Ledger regenerated.Also corrected while I was in there: §4(g) now says why your recomputed A row (0.6043 / +0.0427) differs from this host's own row (0.6050 / +0.0423) in the fourth decimal — your CSV rounds
action_confto four decimals. Recomputing your five columns from the committed file returns your numbers exactly. CI now runs--checkand its five tamper cases as part of the documentation integrity job. No GPU was used, nothing was spent.