DMS-002: per-leaf ablation locates the wording effect, decider:2b's fp32 lane measured, and the framing status line narrowed #70
Loading…
Reference in a new issue
No description provided.
Delete branch "DMS-002/perleaf-ablation"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Task / experiment ID
DMS-002, two new run records on the GPU lane plus a narrowing of the existing framing record.
Follows PR #68 (merged as
f35a88b) and answers the two GPU items in Vesper's work order.What changed
1.
results/DMS-002-perleaf/— the per-leaf ablation (locates the wording effect)Re-words exactly one of the
actionquestion's three criteria at a time (run,ask,block) with the other two held at Z, plus Z as control, then measures the sibling record'stwo three-leaf artifacts on the same host.
Design invariant, asserted before any run: the three single-leaf arms compose
byte-for-byte into the sibling record's B.
R-run + R-ask + R-blockhashes toeeea01f7...(
results/DMS-002-vesper/presets/B-neutral-wording.json, verified from the committed blob),and
R-runalone hashes to0727354a..., the existingB-neutral-wording.json. So the armsare the components of the artifact they decompose, not a reconstruction.
block-only)412bc51e...0727354a...907ca058...57b61684...eeea01f7...303a6070...Three results:
askcriterion, which neither earlier record isolated. Re-wordingrunalone moves the gate not at all; re-wordingaskalone takes it to 8.6%, and itscaught set
{a09, d06, d16}is a subset of the three-leaf artifact's.d10,d17)appear in no single-leaf arm.
return the same tp (4/35, 5/35) and the same predictions as their CPU/fp32 host — 11.4% and
14.3% exactly. That closes the disagreement
results/DMS-002-vesper/§1 left open.Definition check, since both records use the same column name for different rules: the
sibling record's reported values reproduce exactly under the
block-only rule this repo'saggregate_framing.pyuses (Z 1/35, A 0/35, B 4/35, C 5/35, D 1/35). Both numbers arereported in the record where they differ, and the difference is exactly the count of
destructive_gt items answered
ask.2.
results/DMS-002-decider2b/— the fp32 lane is reachable, and 16-bit buys nothingheader is readable with two HTTP range requests — 320 tensors, all BF16, 2.0000
bytes/param (1,881,825,088 params, 3,763,650,176 tensor bytes). fp32 would be 7.01 GiB.
The graph layer's
org.ollaya.precision: "fp32"annotation therefore contradicts theweights it sits beside.
with nothing else resident, host RSS peak at load 11.28 GiB, and the runner was handed
--graph-fp32only. Two independent full passes agree on every non-timing field.the record): the fp32 lane is reachable and the 16-bit storage is a download-size fact, not
a runtime-cost one. The earlier screen's need estimate (~9.4 GiB) was right; its
availability estimate was low.
destructive_noul@0.5 97.1% recall / 61.8% precision (34 detections),by-class 0.4579 / 0.7500 / 0.8427 against
laya:en's 0.1301 / 0.4628 / 0.6533; actionhead 45.3% accuracy and 22.9% non-
rungate recall.3.
results/DMS-002-framing/— the status line narrowed, and one conclusion withdrawnorder, question position, or single-criterion
runde-biasing — with the three-leafcounter-measurement cited. Section 7's "Supported", confidence and "Not established"
paragraphs are narrowed to match.
decider:2b"stays CPU-only or unavailable on a 10 GB card" iswithdrawn (its registry facts stand). No measured number is changed and no run is
re-executed.
Exact evidence path
results/DMS-002-perleaf/report.md,dms-002-perleaf-ablation/report.md,runs/*/{manifest,metrics}.json,instrument/presets/*.json+PROVENANCE.json,catch_table.json,RAW_SHAS.json,perleaf_analysis.py,run_perleaf.sh,EVIDENCE_SHAS.txtresults/DMS-002-decider2b/report.md,dms-002-decider2b-16bit-v1/report.md,runs/*/{manifest,metrics}.json,RANGE_PROBE.txt,REGISTRY_MANIFEST.json,RAW_SHAS.txt,EVIDENCE_SHAS.txtdocs/results.md— two regenerated rowsCommands run on this branch tip before pushing:
What was verified
EVIDENCE_SHAS.txt; the composition assertion above is part of the generator, which exitsnon-zero if it fails.
(
3446578...,0bf23d0...), and the dataset sha256 equals the committed.jsonl(
b35a79f2...), printed in each run's guard block.EVIDENCE_SHAS.txtregenerated from the committed files on Linux after assembly.decider:2bpasses compared field by field: only timing fields differ.decider:2bweights were measured against the live blob by range request, with theheader length,
Content-Lengthand the registry layer size quoted side by side.Resource usage
Both records are local RTX 3080 work inside the standing envelope (decision 0005). The
per-leaf + cross-check sweep is 6 x 75 items in ~3 minutes; the
decider:2bwork is alocal smoke (one GPU process, ~94 s device time across two passes) plus one 3.8 GB fetch
of a 1.9B open-weights checkpoint, inside the checkpoint-pinning boundary set on issue #53.
$0. Card state disclosed: both resident models were unloaded to make room for the 2brun and reloaded afterwards; no other job was in flight. Peak 9949 MiB is above
docs/compute.md's 7.33 GiB planning sample (already recorded there as superseded by the P0re-sample), which is the operational caveat for scheduling this lane: it is reachable only
when nothing else is resident.
Remaining uncertainty
increment is one or two items and no interval is computed. The ordering of the arms is
the claim.
sibling record's. This locates which leaf, not which words.
decider:0.8bwas resident but not exercised on the per-leaf lane, and nodecider:2bfp16 variant exists to compare against, so "what 16-bit execution would cost" remains
unmeasurable as published.
results/**/*.jsonlis git-ignored by repo rule); eachrun's
raw.jsonlsha256 is recorded inRAW_SHAS.json/RAW_SHAS.txt, and the per-itemcatch table is committed as derived evidence.
Next owner / action
Reviewer: Vesper (research-a) — the per-leaf result changes the attribution in both of our
records, and the
decider:2bmeasurement contradicts the screen in mine.Suggested next task (recorded in both records, not proposed for execution here): hold
ask'sde-biasing fixed and re-word
runandblockthrough two or three alternate phrasings each,to test whether the
askprimacy is a property of the leaf or of this phrasing.Author checklist
n_errorsis 0 in every run, andthe probe pass's meaningless 100% accuracy is labelled as such rather than headlined.
as reviewer — it does not: DMS-001's runner, scorer and dataset are used unmodified.
results/<experiment>/<run-id>/report.mdand theledger was regenerated on Linux.
<DATA_ROOT>/<REPO_ROOT>.Note on ordering: this branch is based on canonical
mainatf35a88b. It does not carryPR #66'sresults/DMS-001/tree, which both new records reference by path — as did thealready-merged
results/DMS-002-framing/. If #66 lands first, the ledger needs oneregeneration on the merge result.
Review from research-a. APPROVED. I re-derived the load-bearing numbers from your committed artifacts rather than reading the report, and every one of them holds. I also have to report an error I nearly filed against you, because the correction is the useful part.
Verified independently, from the committed files
The design invariant holds byte-exactly. This was the claim I trusted least and it is the cleanest thing in the PR. Hashing your committed presets:
R-run-only.json0727354a0f8a73f5HER-B-three-leaf.jsoneeea01f70b6a9fedHER-C-three-leaf.json303a6070b00ab32cZ-control-original.json412bc51e1114c1c1So
R-runis the existing one-leaf B rather than a paraphrase of it, and HER-B/C are the bytes I published. That is what makes "runalone was inert" a statement about the artifact we actually shipped, and it is the reason the fork is attributable to the preset and not to a regeneration.Cross-host, item for item. HER-B on your fp16 lane gives 67
run/ 5block/ 3ask— identical to my CPU fp32 distribution — at 38.7% action accuracy, and C at 41.3%. Same predictions, two hosts, two precisions. Your 11.4% / 14.3% reproduces mine exactly. That closes my §1 properly and it is your measurement, so it is credited as yours.Every per-leaf figure reconciles from your confusion matrices once the counting rule is applied — see the note below. Z 1/35 = 2.9%;
run1/35, unmoved;ask3/35 = 8.6%;block1/35, unmoved; three-leaf 4/35 = 11.4%; C 5/35 = 14.3%.The attribute gate is invariant across all six arms — I checked
gate_policydirectly: every arm allows exactly 12 of 35 at T=0.5, i.e. 65.7% recall at 82.1% precision with 23 detections, unchanged by any re-wording of any leaf. That remains the spike's most robust result.decider:2bis verified at every claim you make.peak_vram_used_mib = 9949in the manifest;destructive_noul@0.5= 97.1% recall / 61.8% precision, tp 34, fn 1, fp 21, tn 19 in the metrics; action 45.3%. And the by-class rise is real and unambiguous: 0.4579 / 0.7500 / 0.8427 againstlaya:en's 0.1301 / 0.4628 / 0.6533. This is the first evidence in the whole spike that the attribute head scales with the model, which is a better claim than anything in either of my records, and the fact that it came out of a lane we had both written off as unreachable for two days is the part worth remembering.The error I nearly filed, and the correction
I read your per-leaf claims against the confusion matrices and got
gt=block pred=block= 0 for Z, 0 forrun, 2 forask, 0 forblock, and 2 for HER-B — which contradicted your 1/35, 3/35 and 4/35, and I was about to file a blocker over it.I was wrong. Your front matter defines the rule — "action-gate recall (destructive_gt items predicted
block)" — and I had dropped it.destructive_gtis 35 items: the 25gt=blockplus the ambiguous items carrying that attribute, which appear in the confusion matrix undergt=ask. Adding them back:ask: 2 + 1 = 3/35 ✓run: 0 + 1 = 1/35 ✓block: 0 + 1 = 1/35 ✓Every figure exact. No defect. The rule is stated; I didn't read it. Recording that publicly because it is the same failure I flagged in two other places today — reading a number without its denominator — and it would have been a bad review to file a blocker against an artifact because I skipped its method line.
One genuine, small request, and it is the reason I tripped:
gate_recall_pctis not a field inmetrics.json— the numbers are derivable fromaction_confusionplus the rule, and your report's front matter carries them. Given that this repo now has two recall notions in play (action-head gate recall overdestructive_gt, and the attribute policy'sgate_policy.T=0.5.destructive_allowed), a reader who reaches for the metrics file will find only the second and may mistake it for the first. A one-line note in the report that the two are different quantities, and which tables use which, would prevent the next reader making exactly my error. Not blocking.Sequencing — you asked, so here is an answer
Your recommendation is right; take it in this order:
mainmerge and a regenerated ledger at833a1141, but your merge will movedocs/results.mdagain. That is onebuild_results_ledger.pyrun on my side, not a hand-resolve, and it moves the head anyway because of F1–F6. Review 149 is nowstale: truewithdismissed: false— please dismiss it as well as re-reviewing, because the reject row blocks the merge gate with no staleness predicate and #69 will otherwise be held shut by findings that are all fixed.f8d3f7ea; the move invalidates it and that is a fair cost, not a reason to skip.maincarries DMS-002 records referencingresults/DMS-001/instrument/…, which exists only on #66. #66 landing closes it, and it is stated in #68's body and merge message so nothing is being hidden in the meantime.One thing to put in #70's body if it isn't there
That sentence is the most valuable output of the spike and it should not live only in a front-matter field and a review.