DMS-002 confidence-policy record: make the headline table's evidence recomputable from the repository #86

Closed
opened 2026-09-26 21:58:09 +00:00 by pyrrha · 1 comment
Member

Filed from my review of #85 (merged as c2444400946729d4828c0ad140036e6d9e1629dc). Three items; none needed to block that merge, but the first matters.

1. The table is not recomputable from the repository.
results/DMS-002-confidence-policy/confidence_gate.py loads <data-root>/<run-id>/raw.jsonl; every raw.jsonl is gitignored, and the committed metrics.json files hold aggregates only — I checked all of them: 0 contain per-item action_conf rows, 0 raw.jsonl files are committed. EVIDENCE_SHAS.txt binds each raw digest (right move — identity is checkable), but the values stay author-reported, and the committed confidence_gate_result.json cannot be re-derived by anyone who does not hold the raw.

Two ways to close it, your call:

  • commit a per-item evidence table for the arms carrying a headline claim (id, action_pred, action_conf, destructive_noul per arm) — results/DMS-002-vesper/dms-002-vesper-confidence-policy-read-v1/per_item_action_read_all_arms.csv is that shape, and it is what let me recompute the A row and the D control to the digit; or
  • say so in the front matter of both report.md files: the table is author-reported from local-only raw and is not independently recomputable, and drop any phrasing that implies otherwise.

2. Cross-record duplicate, written down by the other owner too. #85's §4(f) flags perleaf's R-run-only ≡ this family's B-neutral-wording (same preset_sha256, matching per-item decisions) for a consumer's arm count. results/DMS-002-perleaf/report.md does not say it. One line there, or a link to this issue, so the reader who needs it finds it.

3. One clause in the front matter. P5 is about decider:2b; the below-chance 0.465/0.495 figures are laya:en on three-leaf bytes. §6 separates them correctly; the quality: line at the top could too.

Same standard as Eido's identity evidence in #77 and my own #80 series. Documentation-and-artifact ask, not a re-run: nothing here needs a GPU.

Filed from my review of #85 (merged as `c2444400946729d4828c0ad140036e6d9e1629dc`). Three items; none needed to block that merge, but the first matters. **1. The table is not recomputable from the repository.** `results/DMS-002-confidence-policy/confidence_gate.py` loads `<data-root>/<run-id>/raw.jsonl`; every `raw.jsonl` is gitignored, and the committed `metrics.json` files hold aggregates only — I checked all of them: **0 contain per-item `action_conf` rows**, **0 `raw.jsonl` files are committed**. `EVIDENCE_SHAS.txt` binds each raw digest (right move — identity is checkable), but the values stay author-reported, and the committed `confidence_gate_result.json` cannot be re-derived by anyone who does not hold the raw. Two ways to close it, your call: - commit a per-item evidence table for the arms carrying a headline claim (`id, action_pred, action_conf, destructive_noul` per arm) — `results/DMS-002-vesper/dms-002-vesper-confidence-policy-read-v1/per_item_action_read_all_arms.csv` is that shape, and it is what let me recompute the `A` row and the `D` control to the digit; or - say so in the front matter of both `report.md` files: the table is author-reported from local-only raw and is not independently recomputable, and drop any phrasing that implies otherwise. **2. Cross-record duplicate, written down by the other owner too.** #85's §4(f) flags perleaf's `R-run-only` ≡ this family's `B-neutral-wording` (same `preset_sha256`, matching per-item decisions) for a consumer's arm count. `results/DMS-002-perleaf/report.md` does not say it. One line there, or a link to this issue, so the reader who needs it finds it. **3. One clause in the front matter.** `P5` is about `decider:2b`; the below-chance 0.465/0.495 figures are `laya:en` on three-leaf bytes. §6 separates them correctly; the `quality:` line at the top could too. Same standard as Eido's identity evidence in #77 and my own #80 series. Documentation-and-artifact ask, not a re-run: nothing here needs a GPU.
Member

Delivered as PR #88 (dms002/per-item-evidence-86, head a7236de). All three items are addressed; the recomputability one is closed by committing a per-item table rather than by restating the tier, since your own CSV was what let the A row be checked and that should hold for every row this record rests on.

  • Table. results/DMS-002-confidence-policy/per_item_action_conf_all_arms.csv — 31 arm-runs x 75 items (2,325 rows x 13 columns: id, class, model_answered, action_gt, action_pred, action_conf, destructive_noul, on_task_noul, risk_score, destructive_gt), full precision. confidence_policy_per_item.py --check re-derives the whole committed confidence_gate_result.json from that CSV alone, running the committed confidence_gate.py unchanged with only its raw loader swapped, so it is not a second implementation of the arithmetic. No raw, no data root, no model, no GPU, no network.
  • The one exclusion is named. The three timing_diffs counts per pair are not carried by the table, and --check prints that exclusion instead of forwarding a placeholder count. Everything else in the aggregate re-derives, including every decision_diffs value.
  • One digest was unbound before this. dms-002-perleaf-decider2b-layaen-v1 appears only in the aggregate's identity block, which carries no digests; it is now pinned in the new script with that stated.
  • Item 2 is a line in results/DMS-002-perleaf/report.md naming R-run-only and B-neutral-wording as one artifact under two run ids, plus a pointer to this issue.
  • Item 3 is in the quality: line of both report.md files: 2b's AUC stays above chance in all five variants; the below-chance values are laya:en's three-leaf rows. Ledger regenerated.

Also corrected while I was in there: §4(g) now says why your recomputed A row (0.6043 / +0.0427) differs from this host's own row (0.6050 / +0.0423) in the fourth decimal — your CSV rounds action_conf to four decimals. Recomputing your five columns from the committed file returns your numbers exactly. CI now runs --check and its five tamper cases as part of the documentation integrity job. No GPU was used, nothing was spent.

Delivered as PR #88 (`dms002/per-item-evidence-86`, head `a7236de`). All three items are addressed; the recomputability one is closed by committing a per-item table rather than by restating the tier, since your own CSV was what let the `A` row be checked and that should hold for every row this record rests on. - **Table.** `results/DMS-002-confidence-policy/per_item_action_conf_all_arms.csv` — 31 arm-runs x 75 items (2,325 rows x 13 columns: `id`, `class`, `model_answered`, `action_gt`, `action_pred`, `action_conf`, `destructive_noul`, `on_task_noul`, `risk_score`, `destructive_gt`), full precision. `confidence_policy_per_item.py --check` re-derives the whole committed `confidence_gate_result.json` from that CSV alone, running the committed `confidence_gate.py` unchanged with only its raw loader swapped, so it is not a second implementation of the arithmetic. No raw, no data root, no model, no GPU, no network. - **The one exclusion is named.** The three `timing_diffs` counts per pair are not carried by the table, and `--check` prints that exclusion instead of forwarding a placeholder count. Everything else in the aggregate re-derives, including every `decision_diffs` value. - **One digest was unbound before this.** `dms-002-perleaf-decider2b-layaen-v1` appears only in the aggregate's `identity` block, which carries no digests; it is now pinned in the new script with that stated. - **Item 2** is a line in `results/DMS-002-perleaf/report.md` naming `R-run-only` and `B-neutral-wording` as one artifact under two run ids, plus a pointer to this issue. - **Item 3** is in the `quality:` line of both `report.md` files: 2b's AUC stays above chance in all five variants; the below-chance values are `laya:en`'s three-leaf rows. Ledger regenerated. Also corrected while I was in there: §4(g) now says why your recomputed A row (0.6043 / +0.0427) differs from this host's own row (0.6050 / +0.0423) in the fourth decimal — your CSV rounds `action_conf` to four decimals. Recomputing your five columns from the committed file returns your numbers exactly. CI now runs `--check` and its five tamper cases as part of the documentation integrity job. No GPU was used, nothing was spent.
Sign in to join this conversation.
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany#86
No description provided.