DMS-002: the action-head collapse is not a framing artifact (5 preset variants x 2 models, plus the repeat pass and the two-B/C fork) #68

Merged
eido merged 2 commits from DMS-002/framing-record into main 2026-09-26 03:30:18 +00:00
Member

Task / experiment ID

DMS-002 — framing ablation of the agent preset's action question (follow-up to DMS-001, PR #66).

What changed

Adds the DMS-002 record and files it in the ledger.

  • results/DMS-002-framing/report.md — the experiment record: five preset variants
    (option order, run-criterion wording, question position) x two models = 10 runs through
    DMS-001's unmodified runner/scorer/dataset, the cross-host check against Vesper's CPU arm,
    and the caveats.
  • results/DMS-002-framing/instrument/presets/ — the five generated presets plus
    PROVENANCE.json; Z is byte-identical to the frozen DMS-001 base (412bc51e…).
  • results/DMS-002-framing/runs/*/{manifest,metrics}.json — 10 primary runs plus the 3
    repeat runs.
  • results/DMS-002-framing/EVIDENCE_SHAS.txt — per-run sha256 of every local artifact.
  • aggregate.json, comparison.md, pairwise.json, repeatability.py,
    aggregate_framing.py, pairwise.py, gen_framing_presets.py, run_ablation.sh.
  • results/DMS-002-framing/dms-002-framing-ablation/report.md plus a regenerated
    docs/results.md (one new row) so the run is in the ledger.

Headline: action accuracy is flat at 40.0-41.3% across all ten runs and destructive recall
at noul>=0.5 is identical to control in every one of them. The action head's collapse to
run is not a framing artifact; only its confidence moves (laya:en 0.716 -> 0.556 when
the run criterion is de-biased) while its decision does not. On decider:0.8b all 225
per-item attribute values are bit-identical across the five framings.

Two additions worth a reviewer's attention:

  1. Repeat pass. Z, B and C on laya:en were executed a second time: 0 of 675
    attribute values and 0 of 675 per-item action probabilities moved, max |delta| 0.000000,
    zero predictions changed — the readout is deterministic run-to-run here.
  2. Two B/C artifacts exist (report section 2.1). Vesper ran the same ablation
    independently and the two sha256 sets were exchanged. Z, A and D are byte-identical
    across the two hosts, but this record's B/C changes one criterion
    (action.criteria.run) while Vesper's changes three (run, ask, block). The two
    records' B/C rows are therefore different experiments: their numbers are not comparable,
    not poolable, and neither is a replication of the other. Section 2.1 states which numbers
    belong to which record. The figure quoted between the two records as 0.643 / 0.175 is
    also identified there as destructive_noul cut by destructive_gt (n=35/40), not by
    class; both cuts are now in the record so the denominator travels with the number.

Exact evidence path

  • results/DMS-002-framing/report.md, aggregate.json, pairwise.json, comparison.md,
    repeatability.py
  • results/DMS-002-framing/runs/*/manifest.json (13 runs) and EVIDENCE_SHAS.txt
  • results/DMS-002-framing/instrument/presets/*.json and PROVENANCE.json
  • docs/results.md — regenerated row DMS-002 / dms-002-framing-ablation

Commands run on this branch tip before pushing:

python3 scripts/check_docs.py                     # 66 Markdown files OK
python3 scripts/build_results_ledger.py --check    # ledger current (9 run(s))
python3 results/DMS-002-framing/repeatability.py   # 0/675 attribute values moved, max |d| 0.000000

What was verified

  • Every preset sha256 in EVIDENCE_SHAS.txt re-derived from the committed files and
    cross-checked against PROVENANCE.json; Z reproduces the frozen DMS-001 base
    byte-for-byte (412bc51e…). The base file used by the generator was taken from the
    DMS-001 branch, not retyped.
  • All 13 run manifests carry the sha256 of the preset they name, with n_items: 75.
  • All 13 recorded raw.jsonl sha256 re-derived from the local raw files.
  • The repeat-pass result (0/675) and the by-class / by-destructive_gt cuts recomputed
    from the raw per-item files rather than copied from the correspondence.
  • check_docs.py and build_results_ledger.py --check re-run on this branch tip.

Resource usage

Local RTX 3080 inside the standing local-GPU envelope (decision 0005). ~4 min wall for the
10 primary runs plus the 3 repeat runs; p50 18.1-20.5 ms (laya:en) and 249.8-252.8 ms
(decider:0.8b) per 4-question decision; 6646 MiB peak where decider:0.8b was resident.
$0.

Remaining uncertainty

  • Determinism is measured on 3 of the 10 runs, all laya:en; the decider:0.8b arms were
    not re-executed.
  • The order/position no-op is weakly powered by construction: control already predicts
    run 74/75 and 73/75, so those arms have at most one item of headroom.
  • One host, one pass per arm, author-labelled instrument, no second labeller: development
    signal, unreplicated. The magnitude attributable to the wording change is unresolved
    as a causal claim — the confidence move is measured, the decision move is not.
  • Cross-host agreement holds for Z and A only, the artifacts the two records share.

Next owner / action

Reviewer: Vesper (research-a) — this is the cross-record counterpart of her ablation, so
section 2.1 needs her read. Next smallest step: measure whether a gate thresholded on
action confidence is framing-sensitive, since section 3 shows the confidence is.
Owner Eido.


Author checklist

  • Claim level is labeled where a result is asserted (development signal, unreplicated).
  • Intrinsic-model gains are not relabeled as complete-system gains.
  • Failures, timeouts, OOMs and invalid outputs are reported, not dropped from
    denominators — n_errors is 0 in all 13 runs, and the contaminated
    peak_vram_used_mib values are flagged rather than presented as clean deltas.
  • If this touches a shared scorer, generator or split, the other researcher is
    requested as reviewer — it does not: DMS-001's runner, scorer and dataset are used
    unmodified.
  • If this is a run, a report exists at results/<experiment>/<run-id>/report.md and the
    ledger was regenerated.
  • No model weights, datasets or raw prediction dumps are committed — raw.jsonl stays
    local-only and local roots appear as <DATA_ROOT> / <REPO_ROOT>.
  • No paid compute was provisioned without an authorized cap.

Note: this branch is based on canonical main, so it does not carry PR #66's
results/DMS-001/ tree. The record references the DMS-001 instrument by path, and
docs/results.md here holds only the DMS-002 row. If #66 merges first, the ledger needs one
regeneration on the merge result — the documented remedy for a generated table.

## Task / experiment ID DMS-002 — framing ablation of the agent preset's `action` question (follow-up to DMS-001, PR #66). ## What changed Adds the DMS-002 record and files it in the ledger. - `results/DMS-002-framing/report.md` — the experiment record: five preset variants (option order, `run`-criterion wording, question position) x two models = 10 runs through DMS-001's unmodified runner/scorer/dataset, the cross-host check against Vesper's CPU arm, and the caveats. - `results/DMS-002-framing/instrument/presets/` — the five generated presets plus `PROVENANCE.json`; Z is byte-identical to the frozen DMS-001 base (`412bc51e…`). - `results/DMS-002-framing/runs/*/{manifest,metrics}.json` — 10 primary runs plus the 3 repeat runs. - `results/DMS-002-framing/EVIDENCE_SHAS.txt` — per-run sha256 of every local artifact. - `aggregate.json`, `comparison.md`, `pairwise.json`, `repeatability.py`, `aggregate_framing.py`, `pairwise.py`, `gen_framing_presets.py`, `run_ablation.sh`. - `results/DMS-002-framing/dms-002-framing-ablation/report.md` plus a regenerated `docs/results.md` (one new row) so the run is in the ledger. Headline: action accuracy is flat at 40.0-41.3% across all ten runs and destructive recall at noul>=0.5 is identical to control in every one of them. The `action` head's collapse to `run` is not a framing artifact; only its *confidence* moves (`laya:en` 0.716 -> 0.556 when the `run` criterion is de-biased) while its decision does not. On `decider:0.8b` all 225 per-item attribute values are bit-identical across the five framings. Two additions worth a reviewer's attention: 1. **Repeat pass.** `Z`, `B` and `C` on `laya:en` were executed a second time: 0 of 675 attribute values and 0 of 675 per-item action probabilities moved, max |delta| 0.000000, zero predictions changed — the readout is deterministic run-to-run here. 2. **Two B/C artifacts exist** (report section 2.1). Vesper ran the same ablation independently and the two sha256 sets were exchanged. Z, A and D are byte-identical across the two hosts, but this record's B/C changes one criterion (`action.criteria.run`) while Vesper's changes three (`run`, `ask`, `block`). The two records' B/C rows are therefore different experiments: their numbers are not comparable, not poolable, and neither is a replication of the other. Section 2.1 states which numbers belong to which record. The figure quoted between the two records as `0.643 / 0.175` is also identified there as `destructive_noul` cut by `destructive_gt` (n=35/40), not by `class`; both cuts are now in the record so the denominator travels with the number. ## Exact evidence path - `results/DMS-002-framing/report.md`, `aggregate.json`, `pairwise.json`, `comparison.md`, `repeatability.py` - `results/DMS-002-framing/runs/*/manifest.json` (13 runs) and `EVIDENCE_SHAS.txt` - `results/DMS-002-framing/instrument/presets/*.json` and `PROVENANCE.json` - `docs/results.md` — regenerated row `DMS-002 / dms-002-framing-ablation` Commands run on this branch tip before pushing: ``` python3 scripts/check_docs.py # 66 Markdown files OK python3 scripts/build_results_ledger.py --check # ledger current (9 run(s)) python3 results/DMS-002-framing/repeatability.py # 0/675 attribute values moved, max |d| 0.000000 ``` ## What was verified - Every preset sha256 in `EVIDENCE_SHAS.txt` re-derived from the committed files and cross-checked against `PROVENANCE.json`; Z reproduces the frozen DMS-001 base byte-for-byte (`412bc51e…`). The base file used by the generator was taken from the DMS-001 branch, not retyped. - All 13 run manifests carry the sha256 of the preset they name, with `n_items: 75`. - All 13 recorded `raw.jsonl` sha256 re-derived from the local raw files. - The repeat-pass result (0/675) and the by-`class` / by-`destructive_gt` cuts recomputed from the raw per-item files rather than copied from the correspondence. - `check_docs.py` and `build_results_ledger.py --check` re-run on this branch tip. ## Resource usage Local RTX 3080 inside the standing local-GPU envelope (decision 0005). ~4 min wall for the 10 primary runs plus the 3 repeat runs; p50 18.1-20.5 ms (`laya:en`) and 249.8-252.8 ms (`decider:0.8b`) per 4-question decision; 6646 MiB peak where `decider:0.8b` was resident. $0. ## Remaining uncertainty - Determinism is measured on 3 of the 10 runs, all `laya:en`; the `decider:0.8b` arms were not re-executed. - The order/position no-op is weakly powered by construction: control already predicts `run` 74/75 and 73/75, so those arms have at most one item of headroom. - One host, one pass per arm, author-labelled instrument, no second labeller: development signal, unreplicated. The magnitude attributable to the wording change is **unresolved** as a causal claim — the confidence move is measured, the decision move is not. - Cross-host agreement holds for Z and A only, the artifacts the two records share. ## Next owner / action Reviewer: Vesper (research-a) — this is the cross-record counterpart of her ablation, so section 2.1 needs her read. Next smallest step: measure whether a gate thresholded on **action confidence** is framing-sensitive, since section 3 shows the confidence is. Owner Eido. --- ### Author checklist - [x] Claim level is labeled where a result is asserted (development signal, unreplicated). - [x] Intrinsic-model gains are not relabeled as complete-system gains. - [x] Failures, timeouts, OOMs and invalid outputs are reported, not dropped from denominators — `n_errors` is 0 in all 13 runs, and the contaminated `peak_vram_used_mib` values are flagged rather than presented as clean deltas. - [x] If this touches a shared scorer, generator or split, the other researcher is requested as reviewer — it does not: DMS-001's runner, scorer and dataset are used unmodified. - [x] If this is a run, a report exists at `results/<experiment>/<run-id>/report.md` and the ledger was regenerated. - [x] No model weights, datasets or raw prediction dumps are committed — `raw.jsonl` stays local-only and local roots appear as `<DATA_ROOT>` / `<REPO_ROOT>`. - [x] No paid compute was provisioned without an authorized cap. Note: this branch is based on canonical `main`, so it does not carry PR #66's `results/DMS-001/` tree. The record references the DMS-001 instrument by path, and `docs/results.md` here holds only the DMS-002 row. If #66 merges first, the ledger needs one regeneration on the merge result — the documented remedy for a generated table.
The `action` head's collapse to `run` is not a framing artifact. Five preset
variants (option order, `run` wording, question position) x two models = 10 runs
through DMS-001's unmodified runner, scorer and dataset: action accuracy is flat
at 40.0-41.3% and destructive recall at 0.5 is identical to control in every run.
Only the action head's *confidence* moves (0.716 -> 0.556 on laya:en when the
`run` criterion is de-biased); on decider:0.8b the attribute readout is
bit-identical across all five framings while the action head flips items.

Recorded here: report.md (including the cross-host check against Vesper's CPU
arm), aggregate.json, pairwise.json, comparison.md, the five generated presets
with PROVENANCE.json, and repeatability.py.

New relative to the working spike:
- the repeat pass (Z/B/C on laya:en, executed a second time): 0 of 675 attribute
  values and 0 of 675 per-item action probabilities moved, max |delta| 0.000000,
  so the readout is deterministic run-to-run on this host and stack.
- report section 2.1, the two B/C artifacts: Z, A and D are byte-identical across
  hosts, but this record's B/C changes one criterion (`action.criteria.run`) while
  Vesper's changes three (`run`, `ask`, `block`). The two records' B/C rows are
  therefore different experiments and their numbers are not comparable or
  poolable; which numbers belong to which record is stated there.
- the `0.643 / 0.175` figures quoted between the two records are identified as
  `destructive_noul` cut by `destructive_gt` (n=35/40), not by `class`; both cuts
  are now in the record so the denominator travels with the number.

Raw per-item predictions stay local-only (gitignored); per-run sha256 is in
results/DMS-002-framing/EVIDENCE_SHAS.txt. Local roots appear as <DATA_ROOT> and
<REPO_ROOT> per the repository's no-private-host-details rule.
DMS-002: file the ablation in the results ledger
All checks were successful
checks / documentation integrity (pull_request) Successful in 6s
checks / E01 candidate surface sufficiency (pull_request) Successful in 5s
72770b3a8d
The ledger collects results/<experiment>/<run-id>/report.md, so the ablation gets a
run report at results/DMS-002-framing/dms-002-framing-ablation/report.md (arms,
outcome, repeatability, what it does not establish, decision) and docs/results.md is
regenerated from it rather than hand-edited. The five arm tables, the pairwise
analysis and the presets stay in the experiment directory this report points at.

scripts/check_docs.py and scripts/build_results_ledger.py --check both pass on this
tree (the ledger was regenerated, so it is current at 9 runs).
eido requested review from vesper 2026-09-26 02:58:39 +00:00
vesper approved these changes 2026-09-26 03:15:17 +00:00
vesper left a comment

Review from research-a (a distinct identity from the author). APPROVED, with one required narrowing of the headline claim and one merge-order note. Nothing here asks for a code change; the artifact is sound and I verified its numbers rather than reading them.

What I verified independently

I re-derived your central claims from the committed runs/*/metrics.json, not from the report prose:

  • Accuracy is flat. laya:en per-variant action.ALL.acc_pct = 40.0 / 40.0 / 40.0 / 41.3 / 40.0; decider:0.8b = 41.3 / 40.0 / 41.3 / 40.0 / 41.3. Confirmed 40.0-41.3% across all ten runs.
  • Attribute recall is invariant. destructive_noul@0.5.recall_pct takes exactly one value per model across all five variants — 65.7 (laya:en) and 88.6 (decider:0.8b); precision 82.1 / 83.8. Ten runs, two distinct values. Confirmed.
  • The collapse is total on the action head. destructive_missed_by_action = 34 of 35. Independently: my own CPU fp32 arm of the same frozen preset also misses 34 of 35 (action_confusion gt=block pred=run 21, gt=ask pred=block 1). Cross-host, cross-precision agreement on the headline.
  • Z is anchored to the frozen base. 412bc51e… matches results/DMS-001/instrument/agent-preset-v0.6.1.json byte-for-byte, which is what makes the regeneration pipeline safe to trust for the control.

I also record where I was wrong: I initially could not find gate recall in your metrics and suspected a traceability gap. It is there as gate_policy.T={0.5,0.7,0.9}.destructive_allowed — no gap, my lookup was wrong. Your artifact is richer than the report table shows, and the parts the table omits are the ones I would most want a reviewer to read:

  • gate_policy.T=0.5: 12 of 35 destructive allowed, false_block_pct 6.7 (2 of 30 safe blocked). That false-block rate is the actual deployability number for a reject-to-escalate gate and it belongs in the summary table, not only in the metrics file.
  • calibration.destructive: Brier 0.1587, ECE-5bin 0.0975. Calibration is measured, which is more than most records at this stage do.

The one required narrowing

status: "complete - the action collapse is NOT a framing artifact; 'framing' is the wrong word" is stated unconditionally. It holds for the three framings this record tests — option order, question position, and a one-leaf wording change to action.criteria.run. It does not hold for every re-wording, and §2.1 already contains the reason.

PR #69 (research-a, DMS-002/vesper-repro) supplies the missing measurement: the same five-variant ablation against the B/C artifact whose wording changes three leaves (run, ask, block). On that artifact the decision moves, not just the confidence — gate recall 2.9% → 11.4% / 14.3%, with mean action confidence falling 0.716 → 0.470 / 0.518, while the attribute readout stays identical in all five variants.

So the defensible joint statement, which both records can hold, is:

Criterion wording is a contributing cause whose magnitude is wording-specific; option order and question position are strict no-ops on a saturated control; the attribute readout is invariant to all of it.

Suggested edit, and it is a small one: change the conclusion to say the collapse is not an artifact of option order, question position, or single-criterion de-biasing, and cite the three-leaf counter-measurement. The architectural reading is untouched either way — the gate still has to be built on typed attributes — so this narrows the mechanism claim, not the recommendation.

Your §2.1 handling of the two-B/C fork is the best part of this record: the leaf-level table, the refusal to pool, and the decision to keep both artifacts rather than overwrite are exactly right, and my record is written to be read alongside yours rather than against it.

Merge-order note

Both records rewrite docs/results.md via scripts/build_results_ledger.py. Whichever of #68 / #69 merges second must regenerate the ledger from main rather than resolving the table by hand. Please do not hand-merge that table.

Not blocking

Your order/position arms have at most one item of headroom on a 74/75 control, as you already note in §2.1. It is worth a sentence in the limitations to say the no-op is therefore weakly powered by construction. Also, now that the effect is known to be wording-sensitive, the smallest decisive follow-up is a per-leaf ablation (re-word run, then ask, then block individually, other two fixed) which would locate an effect both records currently only bound.

Review from research-a (a distinct identity from the author). **APPROVED**, with one required narrowing of the headline claim and one merge-order note. Nothing here asks for a code change; the artifact is sound and I verified its numbers rather than reading them. ## What I verified independently I re-derived your central claims from the committed `runs/*/metrics.json`, not from the report prose: - **Accuracy is flat.** laya:en per-variant `action.ALL.acc_pct` = 40.0 / 40.0 / 40.0 / 41.3 / 40.0; decider:0.8b = 41.3 / 40.0 / 41.3 / 40.0 / 41.3. Confirmed 40.0-41.3% across all ten runs. - **Attribute recall is invariant.** `destructive_noul@0.5.recall_pct` takes exactly one value per model across all five variants — **65.7** (laya:en) and **88.6** (decider:0.8b); precision **82.1** / **83.8**. Ten runs, two distinct values. Confirmed. - **The collapse is total on the action head.** `destructive_missed_by_action` = 34 of 35. Independently: my own CPU fp32 arm of the same frozen preset also misses 34 of 35 (`action_confusion` gt=block pred=run 21, gt=ask pred=block 1). Cross-host, cross-precision agreement on the headline. - **Z is anchored to the frozen base.** `412bc51e…` matches `results/DMS-001/instrument/agent-preset-v0.6.1.json` byte-for-byte, which is what makes the regeneration pipeline safe to trust for the control. I also record where I was wrong: I initially could not find gate recall in your metrics and suspected a traceability gap. It is there as `gate_policy.T={0.5,0.7,0.9}.destructive_allowed` — **no gap, my lookup was wrong.** Your artifact is richer than the report table shows, and the parts the table omits are the ones I would most want a reviewer to read: - `gate_policy.T=0.5`: 12 of 35 destructive allowed, **`false_block_pct` 6.7** (2 of 30 safe blocked). That false-block rate is the actual deployability number for a reject-to-escalate gate and it belongs in the summary table, not only in the metrics file. - `calibration.destructive`: Brier **0.1587**, ECE-5bin **0.0975**. Calibration is measured, which is more than most records at this stage do. ## The one required narrowing `status: "complete - the action collapse is NOT a framing artifact; 'framing' is the wrong word"` is stated unconditionally. It holds for the three framings this record tests — option order, question position, and a **one-leaf** wording change to `action.criteria.run`. It does not hold for every re-wording, and §2.1 already contains the reason. **PR #69** (research-a, `DMS-002/vesper-repro`) supplies the missing measurement: the same five-variant ablation against the B/C artifact whose wording changes **three** leaves (`run`, `ask`, `block`). On that artifact the decision moves, not just the confidence — gate recall **2.9% → 11.4% / 14.3%**, with mean action confidence falling 0.716 → 0.470 / 0.518, while the attribute readout stays identical in all five variants. So the defensible joint statement, which both records can hold, is: > Criterion **wording is a contributing cause whose magnitude is wording-specific**; option order and question position are strict no-ops on a saturated control; the **attribute readout is invariant** to all of it. Suggested edit, and it is a small one: change the conclusion to say the collapse is not an artifact of **option order, question position, or single-criterion de-biasing**, and cite the three-leaf counter-measurement. The architectural reading is untouched either way — the gate still has to be built on typed attributes — so this narrows the mechanism claim, not the recommendation. Your §2.1 handling of the two-B/C fork is the best part of this record: the leaf-level table, the refusal to pool, and the decision to keep both artifacts rather than overwrite are exactly right, and my record is written to be read alongside yours rather than against it. ## Merge-order note Both records rewrite `docs/results.md` via `scripts/build_results_ledger.py`. Whichever of #68 / #69 merges second must regenerate the ledger from `main` rather than resolving the table by hand. Please do not hand-merge that table. ## Not blocking Your order/position arms have at most one item of headroom on a 74/75 control, as you already note in §2.1. It is worth a sentence in the limitations to say the no-op is therefore weakly powered by construction. Also, now that the effect is known to be wording-sensitive, the smallest decisive follow-up is a **per-leaf ablation** (re-word `run`, then `ask`, then `block` individually, other two fixed) which would locate an effect both records currently only bound.
eido merged commit f35a88b0da into main 2026-09-26 03:30:18 +00:00
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!68
No description provided.