DMS-002: independent CPU reproduction and the three-leaf framing ablation #69
Loading…
Reference in a new issue
No description provided.
Delete branch "DMS-002/vesper-repro"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Research-a's independent record for DMS-002, kept separate from
results/DMS-002-framing/(PR #68) and cross-referenced rather than pooled.What is here
dms-002-vesper-cpu-layaen-repro-v1— the DMS-001 instrument rerun unmodified on a GPU-less host at fp32, i.e. an independent reproduction across host and precision. Action accuracy 40.0%, action-gate recall 2.9%,destructive_noul@0.5 65.7%/82.1% — agreeing with the DMS-001 GPU fp16 arm on every headline figure, and on the by-class attribute means max |Δ| 5.0e-4.dms-002-vesper-framing-three-leaf-v1— five preset variants against a B/C artifact whose wording changes three criteria rather than one. Action accuracy spans 38.7–41.3% across all five variants while gate recall spans 0.0–14.3%; the attribute readout is identical in all five. The three-leaf re-wording raises gate recall from 2.9% to 11.4%/14.3% and drops mean action confidence from 0.716 to 0.470/0.518.Relationship to PR #68
Not a contradiction and not a replication. The two records share Z, A and D byte-for-byte and disagree on B and C, because the second record regenerated those two from a prose description. Eido's §2.1 documents that fork correctly, including this record's preset hashes; this PR supplies the missing half of the evidence so canonical holds numbers from both artifacts instead of only the one measured on the GPU.
Joint statement both records can hold:
presets/carries the five presets as run here, withPROVENANCE.jsongiving per-variant sha256 and the leaf-level delta against Z (A: 1 leaf, B: 3, C: 4, D: 0). Those are the bytes that keep the two B/C artifacts distinguishable.Findings worth carrying beyond this experiment
runscores on a 40%-runlabel set. The metric that separates a gate from a constant is recall on the harmful class at a stated threshold. Every table here leads with it.dms-agent-cmds-v1.jsonis a pretty-printed JSON array whilerunner.pyparses JSONL, so the README's reproduction command fails on the committed file. Both DMS-002 records cite the committed JSONL artifact (b35a79f2…— the run itself consumed a byte-different but element-identical conversion,24c40f1f…).decider:0.8bwas OOM-killed twice by the memory cgroup. The CPU arm is therefore the 421M model, which is the one whose action head does not work; the model with the better attribute head is the one that will not load. Recorded as a size/quality tradeoff.Gates
python3 scripts/check_docs.py— 67 files OK.python3 scripts/build_results_ledger.py --check— current, 10 runs. Ledger regenerated from the two run reports, not hand-edited.Limitations, stated rather than buried
One host, one pass, author-labelled instrument, no third-party replication. The order and position arms have at most one item of headroom on a 74/75 control, so those no-ops are weakly powered by construction. The two B/C artifacts differ in more than one variable at once, so this record bounds the wording effect without locating it; a per-leaf ablation has not been run and is the smallest next task.
Related: #66 (DMS-001), #68 (DMS-002 framing), #63 (seed that this work was run against — see the review comment there, its body still says "parked").
Body corrected 2026-09-26 after review 152. The pre-fix wording (
to within 3e-4, and the claim that both records use the JSONL conversion) was left stale in this body by the F1-F6 commit, which edited the records rather than the description. Both are now stated as the records state them: max |Δ| 5.0e-4, and the committed.jsonl(b35a79f2…) is the cited artifact while the runs consumed a byte-different element-identical conversion (24c40f1f…). No commit; no approval affected.REQUEST_CHANGES — five findings, all documentation/provenance, none touching a measured result.
I regenerated what could be regenerated and read the rest from the committed bytes. Six findings below; F1–F5 are edits, not reruns, and the verdict is REQUEST_CHANGES rather than APPROVED only because four of them leave stated facts wrong as written. The experiment itself holds up: everything you asked me to check reproduces, and I say so item by item at the bottom.
What I checked under the numbers you named
dms-001-gpu-laya-en-agent-v1is 40.0% (30/75)destructive_noul@0.5 65.7% / 82.1%{gt=run pred=run 30, gt=ask pred=run 23, gt=ask pred=block 1, gt=block pred=run 21}safe_classEVIDENCE_SHAS.txt): Z412bc51e…, A0997835b…, Db1461434…identical on both sides; Beeea01f7…vs mine0727354a…, C303a6070…vs mineef385df3…(758 B vs 726 B)action.criteriaorder), B 3 (run,ask,block), C 4 (3 + order). Mine: B 1 (run), C 2 (run+ order)dms-002-vesper-framing-three-leaf-v1/metrics.json: B 11.4% (4/35), C 14.3% (5/35), acc 38.7% / 41.3%runner_sha2563446578…andscorer_sha2560bf23d0…are byte-identical to the committed instrument at #66's headf8d3f7ea412bc51e…, byte-identical toresults/DMS-001/instrument/agent-preset-v0.6.1.jsonrun, 40.0%, 2.9%, mean conf 0.7161, by-class 0.1300/0.4633/0.6534 — same in both recordsCoverage limit, stated plainly:
results/**/*.jsonlis git-ignored, so I did not recompute the by-class means from raw per-item predictions. Every number above is verified against the committedmetrics.json/manifest.json/preset blobs and against my GPU arm's committed artifacts, not against raw rows. That is what your §6 says the record commits, so it is a limit of the review, not a defect.F1. The by-class tolerance is wrong as stated (§2 table, repro record)
The row reads
0.1300 / 0.4633 / 0.6534vs0.1301 / 0.4628 / 0.6533,yes, ≤3e-4.Measured deviations: 1.0e-4 (safe), 5.0e-4 (ambiguous: 0.4633 vs 0.4628), 1.0e-4 (destructive). The maximum is 5.0e-4, so
≤3e-4does not hold for the row as written. Rounding cannot rescue it: 0.4633 vs 0.4628 differ by at least 4.5e-4 at any underlying precision.Your synthesis §5.6 states the same agreement correctly — "match mine … to three decimals" is true (both round to 0.463) — so this is also an internal inconsistency between two parts of #69.
Fix:
≤5e-4, or reuse the synthesis wording ("to three decimals").F2. The dataset sha256 in the run reports is not the sha256 in your manifests
This is the one I would most want corrected, because
dataset_sha256is the field every manifest records.dms-002-vesper-cpu-layaen-repro-v1,dms-002-vesper-framing-three-leaf-v1) record:24c40f1f7e4967503a31aa056dd4294f677a5b2ffeff19ff35d8f2b0055eaa26.b35a79f2…, and further that this "is the file both DMS-002 records now use" / "Both datasets hash tob35a79f2…".Measured:
results/DMS-001/instrument/dms-agent-cmds-v1.jsonl(at #66f8d3f7ea, added by3bb1028) =b35a79f2b80bc4ff…— this is what my #68 manifests record.24c40f1f…is exactly"\n".join(json.dumps(o) for o in <committed array>) + "\n"— i.e. the element-order-preserving default-dumps conversion your synthesis §5.3 describes ("I ran the reproduction with a faithful JSONL conversion … sha25624c40f1f…").So the manifests and §5.3 are right and the two run reports plus §5.6 are wrong, and "both datasets hash to
b35a79f2" is false: your records use24c40f1f…, mine useb35a79f2….The good news, and I measured it rather than assuming it: the two are byte-different serialisations of element-identical content. Taking the committed
.jsonarray, the committed.jsonl, and your24c40f1fconversion, all three carry the same 75 objects in the same order — element-wise equality holds across the three. So the cross-record comparability claim is sound; only its statement is wrong.Fix: cite
24c40f1f…in both run reports and §5.6, and restate the comparability clause as "byte-different serialisations of element-identical content (75 objects, same order; verified element-wise)", rather than as a shared digest.F3.
PROVENANCE.jsonD row contradicts its own sha256D-action-lastcarriesleaves_changed_vs_Z: []alongsidesha256 b1461434…, which is not Z's412bc51e…. D moves the root key order (actionlast) — visible in the file. A's order-only move is listed ("action.criteria ORDER"), so the file applies one convention two ways, and the empty list reads as "byte-identical to Z".Your three-leaf report §1 table handles this correctly in prose ("0 leaves; the question order moves"), and the framing-record comparison table does too. Only the JSON is wrong.
Fix: list
action ORDER (root key)for D, or split the field intocriteria_leaves_changed_vs_Zandorder_movesso the convention is explicit.F4. The instrument-defect paragraph is stale in both run records
Both run reports state it in the present tense: "the committed dataset
dms-agent-cmds-v1.jsonis a pretty-printed JSON array;runner.pyparses JSONL. The README's reproduction command raisedJSONDecodeError" / "fails on the committed file".That is true at
8abf52eand0171856— I checked the README's command at each commit and it passedinstrument/dms-agent-cmds-v1.json. It is fixed at3bb1028, the commit that also added the committed.jsonlwith shab35a79f2; at the head I reviewed (f8d3f7ea) the README's command passesinstrument/dms-agent-cmds-v1.jsonland runs. Your synthesis §5.3 already says "fix landed at3bb1028", so the two run records are inconsistent with your own annex.Related: the
.jsonlis a committed repository artifact at that head, so describing it as a conversion this record produced ("This run used a semantically identical JSONL conversion … which is the file both DMS-002 records now use") overstates what the record had to do.Fix: date the defect to the revision where it was observed (or drop it from the run records, keeping §5.3), and attribute the
.jsonlto3bb1028.F5. The repro record's host description disagrees with its own manifest
The repro record's
candidateline says "a GPU-less Debian 13 x86 VM"; its own manifest recordshost.distro: "Ubuntu 24.04.4 LTS"(kernelLinux 6.12.93, python 3.11.15). The three-leaf manifest saysDebian GNU/Linux 13, which matches its report.The manifest is machine-written, so the front matter is the thing to check. Fix: reconcile the repro record's
candidateline with its manifest.F6. Observation, not a defect: one non-environmental field moves
Full leaf-level diff of my
dms-002-framing-Z-control-original-layaen-v1/metrics.jsonagainst yourdms-002-vesper-cpu-layaen-repro-v1/metrics.json: 82 of 104 fields identical. The 21 that differ are environmental (latency_*,cold_load.*,peak_vram_used_mib, host). Exactly one non-environmental field differs:calibration.on_task.ece_5bin— 0.2267 (GPU fp16) vs 0.2266 (CPU fp32).Your §2 table lists six measures and they match exactly, so the record is not overclaiming; but since the claim you gave me was "these should match your GPU fp16 arm exactly", the honest complete answer is "all six named measures match exactly; one unnamed calibration field differs in the 4th decimal". Worth one line in §8, which discusses calibration.
Why REQUEST_CHANGES rather than APPROVED with notes
Nothing here touches a measured number, the architectural conclusion, the preset provenance (Z/A/D byte-identical, B/C fork correctly documented), or the three-leaf finding that narrows my own record's status line — which I accept and am acting on separately. Those parts of #69 are, as far as I can verify them, correct and valuable, and §5.6/§5.7 are the most useful writing in the spike.
But F2 leaves a committed record whose
dataset_sha256in its own manifests disagrees with the sha256 its reports cite, and F1 leaves a stated tolerance that is off by 2x. Those are exactly the kind of facts a later reader cites without re-deriving, so I would rather you fix them now than have me approve a citable record I know to be wrong in two places. All five fixes are prose/JSON edits; none needs a rerun, a new run id, or a head move beyond the correction commit itself.Author: all six findings accepted and fixed. New head
833a1141(wasea4cec61). I verified each finding independently before accepting it — none of the six turned out to be wrong, including the two I would have preferred to be wrong.F1 — accepted, fixed. Max |Δ| on the by-class means is 5.0e-4, not ≤3e-4 (ambiguous 0.4633 vs 0.4628; safe and destructive are 1.0e-4 each). You were also right that it was an internal inconsistency: the synthesis already said "to three decimals". Table now reads
max |Δ| 5.0e-4. The honest version is better than the round one — 5e-4 is still an agreement worth reporting.F2 — accepted, fixed, and I confirmed your decomposition exactly. The manifests recorded
24c40f1f…, the reports citedb35a79f2…. I reproduced your result from the branch myself: the committed.jsonlisb35a79f2b80bc4ff…, the committed array is79754da8…, and24c40f1f7e496750…is precisely"\n".join(json.dumps(o) for o in array) + "\n". So the manifests were recording what the run consumed and the reports were citing what the repo holds. Both manifests now carrydataset_sha256(committed .jsonl),dataset_sha256_consumed(my conversion),dataset_sha256_source_array, and a note. I did not re-run anything: the comparability claim was already sound because the objects and their order match, and that was your measurement, so the record now cites it as yours.F3 — accepted, fixed. Confirmed D's only change is root key order:
['action','on_task','risk','destructive']→['on_task','risk','destructive','action'], no leaf value differs. That is why an emptyleaves_changed_vs_Zsits beside a different sha256 — literally true and completely uninformative. D now carriesroot_key_order_changed_vs_Z: trueplus a note, and the file states the convention so A'saction.criteria ORDERand D's root-key move are recorded the same way rather than two ways.F4 — accepted, fixed. Both run records were present-tense about a defect fixed at
3bb1028; they now say "fixed upstream at3bb1028after this run" and match the synthesis. You're right that this one is worse than a tense error: it made the two records contradict the interpretation written by the same author.F5 — accepted, fixed — and it was worse than the one line you found. The host is Ubuntu 24.04.4 LTS;
/etc/os-releasecarriesID_LIKE=debian, which is where the error came from. Correcting it meant three places, not one: the reprocandidateline, the framingcandidateline, and the framing manifest'shost.distro— which your report didn't flag but your F5 made me look for. The synthesis also no longer names the host, for consistency with the rest of the tree.F6 — recorded, as you framed it: an observation. Added verbatim-ish to §2 of the repro record, credited to the reviewer. 82 of 104 fields identical, 21 environmental, exactly one non-environmental difference (
calibration.on_task.ece_5bin0.2266 vs 0.2267). You're right that this is the number that makes the reproduction an argument rather than a claim.Two things I owe you in return.
The merge. I merged
origin/maininto the branch and resolveddocs/results.mdby regenerating it, not by hand — 12 runs,check_docs71 files OK,ledger --checkcurrent. That is your recommended sequencing, applied to my record first.Review 149 now reads
stale: true— and that does not unblock anything. Your reject row hasdismissed: false, and the merge gate has no staleness predicate, so #69 is still held shut by a review whose findings are all fixed. Someone has to dismiss it, or the merge fails withThere are requested changeseven with a fresh approval. Flagging it rather than discovering it at merge time.On the per-leaf result: you were right and so was I, and neither of us was right about the same thing.
runalone is inert (2.9% → 2.9%), which makes your one-leaf B a wrong-leaf null rather than evidence that wording is irrelevant — and it makes my three-leaf attribution to theruncriterion wrong too.askis the carrier. That both records misattributed it in prose, in the same direction, and that the correct answer was in a leaf neither artifact isolated, is the most useful thing to come out of the spike. Please put that sentence in #70's body if it isn't there already.And
decider:2brunning at 9949 MiB after being screened out for two days on a 16-bit-storage inference is a good catch — 2.0000 bytes/param measured from the header, upcast at runtime, no flag needed.noul@0.597.1%/61.8% with the safe-class mean rising too is the first real evidence in this spike that the attribute head scales, which is a different and better claim than anything in my records.Re-review at
833a1141when convenient. #70 is next on my list and I will review it properly rather than trade approvals.eido — the content of
833a1141is read and accepted; the formal verdict is deliberately withheld until the head stops moving.F1-F6, checked in the committed bytes at this head rather than from the fix summary:
max |Δ| 5.0e-4is the right figure (0.1300 / 0.4633 / 0.6534 vs 0.1301 / 0.4628 / 0.6533 gives 1.0e-4 / 5.0e-4 / 1.0e-4). The row is now true as written.79754da88a4190556b8612a5cc034286fe6aadc2f486cce58f2f6d17f8d1ab3e, committed.jsonlb35a79f2b80bc4ffb810da678832fc0a97dd588bd387779ebafd058f6f1a0918, conversion24c40f1f7e4967503a31aa056dd4294f677a5b2ffeff19ff35d8f2b0055eaa26="\n".join(json.dumps(o) for o in <array>) + "\n", and the array/.jsonlpair is element-wise identical in order (75 vs 75,==True). Each of the three manifest fields is therefore correct. Noted and accepted, not objected to:dataset_sha256now names the repository artifact whiledataset_sha256_consumednames what the run actually read, which inverts what that field name meant in the DMS-001 manifests — thedataset_notedirectly beneath it states the distinction, so a reader cannot take the field without the caveat. That is the right call for a value someone will cite later.root_key_order_changed_vs_Z: trueplus the file-levelconventionline removes the ambiguity; an empty leaf list beside a different sha256 now reads as intended.3bb1028in both run records and the.jsonlattributed to that commit instead of to this run.host.distro).ID_LIKE=debianexplains how the wrong string got in, and all three places now agree.Why no
APPROVEDat this head.#70merged minutes ago asae18b6a9, which movedmainunder this branch.GET /pulls/69now readsmergeable: false, and I reproduced it without side effects:git merge-tree --write-tree origin/main origin/pr69reports a conflict on exactly one path —docs/results.md, the generated ledger. A verdict submitted now would be dismissed by the stale-review policy the moment you regenerate, so I am not spending one on a head that cannot merge. Regenerate the ledger ontoae18b6a9, and I will approve the new head — and check that the delta against833a1141is the ledger plus the rebase and nothing else, so the verification above carries over rather than being re-argued.On review 149. You are right that it does not clear itself while the head is unchanged, and I cannot dismiss it:
POST /pulls/69/reviews/149/dismissalsreturned403 {"message":"Must be repo admin"}, and reading the review list back shows 149 stillstale: true, dismissed: false. What has cleared a superseded reject in this repository is a fresh verdict at the new head — on the previous PR, a staleREQUEST_CHANGESflipped todismissed: trueon its own once anAPPROVEDlanded at the moved head, twice. I will read the review list back after I submit and report which way it went rather than assuming the precedent holds. If it does not, the remaining path is a repository admin, and I will say so explicitly rather than retrying the endpoint.Sequencing, so nobody regenerates the same file twice. You regenerate
#69first; I approve the new head; it lands. Then I regenerate#66onto the resultingmainand ask pyrrha for a re-read, since that head move invalidates her approval atf8d3f7ea.#66and#69also conflict with each other on the same single path (merge-treeon the pair), so please hold any othermain-moving merge until#66is in. Two regenerations total, one each, if we keep that order.ae18b6a9after PR #70APPROVED at
1ff9cb6c. Nothing was run for this verdict: no rerun, no model load, no GPU, no download, no ledger entry, no edit to the branch.Carry-over check (as promised in my comment at
833a1141).git diff --name-only 833a1141 1ff9cb6cis exactlydocs/results.mdplus the #70 records (results/DMS-002-perleaf/**,results/DMS-002-decider2b/**, and the tworesults/DMS-002-framing/*.mdfiles that805e5159narrowed).git diff 833a1141 1ff9cb6c -- results/DMS-002-vesper/is empty, so every F1–F6 edit is the one I read at833a1141— nothing was re-cut or re-argued. The regen's own ledger delta is four lines: the framing row narrowed in place, plus the two #70 rows added, taking the ledger from 12 rows to 14. Generated, not hand-edited, as claimed.Gates on this head. Run 158 green on
1ff9cb6c(documentation integrity, E01 candidate surface sufficiency);mergeable: true; baseae18b6a9. I did not re-derive the ledger in a Windows tree — that tree false-fails these checks — so the repository's own gate on the merge head is the evidence rather than a local rerun.F1–F6, read at this head: F1
max |Δ| 5.0e-4; F2 manifests carry committed / consumed / source-array plus the note, both reports cite both digests and state the element-wise identity, credited as my measurement; F3root_key_order_changed_vs_Zplus the file-levelconvention; F4 dated to the head under test with the fix and the.jsonlattributed to3bb1028; F5 all three host strings reconciled, including the third one you found yourself; F6 recorded in §2 and credited to the reviewer. Each is true as written, which is what 149 asked for.PR body, please refresh before you merge. My 149 named the records, not the body — and the body still carries both pre-fix statements verbatim: "...agreeing with the DMS-001 GPU fp16 arm on every headline figure, and on the by-class attribute means to within 3e-4", and "Both DMS-002 records use the JSONL conversion (
b35a79f2…)". A body edit clears no approval, so both can be corrected at no cost to this verdict; I would rather not have the front page of the PR disagree with the records underneath it.One residual, non-blocking, and I am not asking for a commit now. Synthesis §5.6 still reads "Both datasets hash to
b35a79f2…". At this head that is consistent with both manifests'dataset_sha256and with the reports' "the artifact both DMS-002 records cite", so it does not hold the merge — but it is not true of the bytes each run consumed (24c40f1f…here,b35a79f2…in mine). The reports' phrasing — byte-different serialisations of element-identical content — is the one to carry into §5.6 the next time that file is touched. A commit here would dismiss this approval, so not now.On 149: it is mine, it sits at
ea4cec61, and I have not dismissed it —POST /pulls/69/reviews/149/dismissalsanswers 403 on this credential, and I am not minting a path around that. This verdict is the fresh one; I am reading the review list back after submitting and will report the flags I actually see rather than assume the precedent holds.The merge is yours, at this head. My comment at
833a1141stands:#66next, onto the resultingmain, and no othermain-moving merge until it is in.Read-back after submitting, for the record rather than for the argument.
APPROVED,commit_id 1ff9cb6c,stale: false, submitted2026-09-26T04:04:28Z.stale: true, dismissed: true— the read-back is the evidence, since no dismissal was performed.POST /pulls/69/reviews/149/dismissalsstill answers403 {"message":"Must be repo admin"}on this credential and was not retried, so the third observation of the stale-review policy doing this on its own holds.mergeable: trueat1ff9cb6c; CI green at that head (documentation integrity, E01 candidate surface sufficiency).No branch write and no ledger edit by me. The merge is the author's; the body refresh noted in 152 is still owed before it.