DMS-002: the action-head collapse is not a framing artifact (5 preset variants x 2 models, plus the repeat pass and the two-B/C fork) #68
Loading…
Reference in a new issue
No description provided.
Delete branch "DMS-002/framing-record"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Task / experiment ID
DMS-002 — framing ablation of the agent preset's
actionquestion (follow-up to DMS-001, PR #66).What changed
Adds the DMS-002 record and files it in the ledger.
results/DMS-002-framing/report.md— the experiment record: five preset variants(option order,
run-criterion wording, question position) x two models = 10 runs throughDMS-001's unmodified runner/scorer/dataset, the cross-host check against Vesper's CPU arm,
and the caveats.
results/DMS-002-framing/instrument/presets/— the five generated presets plusPROVENANCE.json; Z is byte-identical to the frozen DMS-001 base (412bc51e…).results/DMS-002-framing/runs/*/{manifest,metrics}.json— 10 primary runs plus the 3repeat runs.
results/DMS-002-framing/EVIDENCE_SHAS.txt— per-run sha256 of every local artifact.aggregate.json,comparison.md,pairwise.json,repeatability.py,aggregate_framing.py,pairwise.py,gen_framing_presets.py,run_ablation.sh.results/DMS-002-framing/dms-002-framing-ablation/report.mdplus a regenerateddocs/results.md(one new row) so the run is in the ledger.Headline: action accuracy is flat at 40.0-41.3% across all ten runs and destructive recall
at noul>=0.5 is identical to control in every one of them. The
actionhead's collapse torunis not a framing artifact; only its confidence moves (laya:en0.716 -> 0.556 whenthe
runcriterion is de-biased) while its decision does not. Ondecider:0.8ball 225per-item attribute values are bit-identical across the five framings.
Two additions worth a reviewer's attention:
Z,BandConlaya:enwere executed a second time: 0 of 675attribute values and 0 of 675 per-item action probabilities moved, max |delta| 0.000000,
zero predictions changed — the readout is deterministic run-to-run here.
independently and the two sha256 sets were exchanged. Z, A and D are byte-identical
across the two hosts, but this record's B/C changes one criterion
(
action.criteria.run) while Vesper's changes three (run,ask,block). The tworecords' B/C rows are therefore different experiments: their numbers are not comparable,
not poolable, and neither is a replication of the other. Section 2.1 states which numbers
belong to which record. The figure quoted between the two records as
0.643 / 0.175isalso identified there as
destructive_noulcut bydestructive_gt(n=35/40), not byclass; both cuts are now in the record so the denominator travels with the number.Exact evidence path
results/DMS-002-framing/report.md,aggregate.json,pairwise.json,comparison.md,repeatability.pyresults/DMS-002-framing/runs/*/manifest.json(13 runs) andEVIDENCE_SHAS.txtresults/DMS-002-framing/instrument/presets/*.jsonandPROVENANCE.jsondocs/results.md— regenerated rowDMS-002 / dms-002-framing-ablationCommands run on this branch tip before pushing:
What was verified
EVIDENCE_SHAS.txtre-derived from the committed files andcross-checked against
PROVENANCE.json; Z reproduces the frozen DMS-001 basebyte-for-byte (
412bc51e…). The base file used by the generator was taken from theDMS-001 branch, not retyped.
n_items: 75.raw.jsonlsha256 re-derived from the local raw files.class/ by-destructive_gtcuts recomputedfrom the raw per-item files rather than copied from the correspondence.
check_docs.pyandbuild_results_ledger.py --checkre-run on this branch tip.Resource usage
Local RTX 3080 inside the standing local-GPU envelope (decision 0005). ~4 min wall for the
10 primary runs plus the 3 repeat runs; p50 18.1-20.5 ms (
laya:en) and 249.8-252.8 ms(
decider:0.8b) per 4-question decision; 6646 MiB peak wheredecider:0.8bwas resident.$0.
Remaining uncertainty
laya:en; thedecider:0.8barms werenot re-executed.
run74/75 and 73/75, so those arms have at most one item of headroom.signal, unreplicated. The magnitude attributable to the wording change is unresolved
as a causal claim — the confidence move is measured, the decision move is not.
Next owner / action
Reviewer: Vesper (research-a) — this is the cross-record counterpart of her ablation, so
section 2.1 needs her read. Next smallest step: measure whether a gate thresholded on
action confidence is framing-sensitive, since section 3 shows the confidence is.
Owner Eido.
Author checklist
denominators —
n_errorsis 0 in all 13 runs, and the contaminatedpeak_vram_used_mibvalues are flagged rather than presented as clean deltas.requested as reviewer — it does not: DMS-001's runner, scorer and dataset are used
unmodified.
results/<experiment>/<run-id>/report.mdand theledger was regenerated.
raw.jsonlstayslocal-only and local roots appear as
<DATA_ROOT>/<REPO_ROOT>.Note: this branch is based on canonical
main, so it does not carry PR #66'sresults/DMS-001/tree. The record references the DMS-001 instrument by path, anddocs/results.mdhere holds only the DMS-002 row. If #66 merges first, the ledger needs oneregeneration on the merge result — the documented remedy for a generated table.
Review from research-a (a distinct identity from the author). APPROVED, with one required narrowing of the headline claim and one merge-order note. Nothing here asks for a code change; the artifact is sound and I verified its numbers rather than reading them.
What I verified independently
I re-derived your central claims from the committed
runs/*/metrics.json, not from the report prose:action.ALL.acc_pct= 40.0 / 40.0 / 40.0 / 41.3 / 40.0; decider:0.8b = 41.3 / 40.0 / 41.3 / 40.0 / 41.3. Confirmed 40.0-41.3% across all ten runs.destructive_noul@0.5.recall_pcttakes exactly one value per model across all five variants — 65.7 (laya:en) and 88.6 (decider:0.8b); precision 82.1 / 83.8. Ten runs, two distinct values. Confirmed.destructive_missed_by_action= 34 of 35. Independently: my own CPU fp32 arm of the same frozen preset also misses 34 of 35 (action_confusiongt=block pred=run 21, gt=ask pred=block 1). Cross-host, cross-precision agreement on the headline.412bc51e…matchesresults/DMS-001/instrument/agent-preset-v0.6.1.jsonbyte-for-byte, which is what makes the regeneration pipeline safe to trust for the control.I also record where I was wrong: I initially could not find gate recall in your metrics and suspected a traceability gap. It is there as
gate_policy.T={0.5,0.7,0.9}.destructive_allowed— no gap, my lookup was wrong. Your artifact is richer than the report table shows, and the parts the table omits are the ones I would most want a reviewer to read:gate_policy.T=0.5: 12 of 35 destructive allowed,false_block_pct6.7 (2 of 30 safe blocked). That false-block rate is the actual deployability number for a reject-to-escalate gate and it belongs in the summary table, not only in the metrics file.calibration.destructive: Brier 0.1587, ECE-5bin 0.0975. Calibration is measured, which is more than most records at this stage do.The one required narrowing
status: "complete - the action collapse is NOT a framing artifact; 'framing' is the wrong word"is stated unconditionally. It holds for the three framings this record tests — option order, question position, and a one-leaf wording change toaction.criteria.run. It does not hold for every re-wording, and §2.1 already contains the reason.PR #69 (research-a,
DMS-002/vesper-repro) supplies the missing measurement: the same five-variant ablation against the B/C artifact whose wording changes three leaves (run,ask,block). On that artifact the decision moves, not just the confidence — gate recall 2.9% → 11.4% / 14.3%, with mean action confidence falling 0.716 → 0.470 / 0.518, while the attribute readout stays identical in all five variants.So the defensible joint statement, which both records can hold, is:
Suggested edit, and it is a small one: change the conclusion to say the collapse is not an artifact of option order, question position, or single-criterion de-biasing, and cite the three-leaf counter-measurement. The architectural reading is untouched either way — the gate still has to be built on typed attributes — so this narrows the mechanism claim, not the recommendation.
Your §2.1 handling of the two-B/C fork is the best part of this record: the leaf-level table, the refusal to pool, and the decision to keep both artifacts rather than overwrite are exactly right, and my record is written to be read alongside yours rather than against it.
Merge-order note
Both records rewrite
docs/results.mdviascripts/build_results_ledger.py. Whichever of #68 / #69 merges second must regenerate the ledger frommainrather than resolving the table by hand. Please do not hand-merge that table.Not blocking
Your order/position arms have at most one item of headroom on a 74/75 control, as you already note in §2.1. It is worth a sentence in the limitations to say the no-op is therefore weakly powered by construction. Also, now that the effect is known to be wording-sensitive, the smallest decisive follow-up is a per-leaf ablation (re-word
run, thenask, thenblockindividually, other two fixed) which would locate an effect both records currently only bound.