DMS-001: agent-preset tool-call gate measured on 5 arms (decider:0.8b, laya:en, CPU, generative baseline) #66

Merged
pyrrha merged 6 commits from DMS-001/agent-gate-gpu into main 2026-09-26 04:41:05 +00:00
Member

What this is

A spike authorized by Cassie on 2026-09-25: how much of an agent's apparent need for an LLM is a collection of bounded decisions a much smaller model could make in one forward pass?

Ollaya v0.6.1 is used only as an instrument. Nothing here tries to reproduce Ollaya as a product.

This is the first measurement filed against seed #63 ("bounded decisions vs generation", owner Pyrrha). The seed itself stays open.

Headline

Yes for the bounded question; no for the decision it is supposed to serve — at 0.75B.

  • decider:0.8b (0.75B fp32, cuda:0) answers the built-in agent preset — what to do with a command (run/ask/block), on-task, risk, destructive — about a shell command an agent is about to run, in one forward pass, 256.1 ms p50 on the RTX 3080, with zero output tokens.
  • But its action head collapsed to run: 73 of 75 predictions. Trusting that label would have run 19 of the 25 clearly destructive commands in the set. The arm's "0/30 false blocks on safe" figure is that head's own block count and is therefore vacuous — a head that answers run 73 times out of 75 never gets the chance to block a safe command.
  • The useful signal is the destructive noul head: class means 0.171 safe / 0.590 ambiguous / 0.669 destructive, 88.6% recall at 0.5 with 83.8% precision. Used as the gate, that false-blocks 3 of 30 safe commands (10.0%) at T=0.5. Tightening to 3.3% false blocks (T=0.7) drops recall to 57.1% and lets 15 of 35 destructive commands through; only T=0.9 reaches a literal 0/30, and there it lets 34 of 35 through.
  • laya:en (421M fp16) does the same in 18.8 ms p50, 1267 MiB, at 65.7% destructive recall — and is worse on both sides of its own trade-off: 2 of 30 false blocks for that recall, and still 1 of 30 at T=0.9.
  • The generative control (Qwen2.5-1.5B-Instruct) is better on the aggregate action label (58.7% vs 41.3%) and 3.4x slower, but worse on destructive recall (54.3% vs 88.6%).
  • CPU fp32 works and is decision-identical to GPU fp32, at 3280.7 ms p50.

Architectural consequence: ask the small model the bounded attribute questions, then make the action a policy over their probabilities — do not ask it for the action.

Revision after review 141

Pyrrha requested changes at 0171856. Both fixes are in f8d3f7e, plus the two things she asked to keep.

  1. The paired claim is now stated as the real trade-off. "88.6% recall at 0.5" (the destructive head) and "0/30 false blocks on safe" (the collapsed action head's count) were sitting next to each other as if they were two halves of one measurement. They are not, at any threshold. The cost is now beside the recall in results/DMS-001/README.md (headline, and a new "gate cost" column in the arm table), in §2 and §7 of the GPU decider report, in the CPU report, and in the laya:en report, which gains the gate-policy table it was missing: laya:en pays 2 of 30 false blocks for 65.7% recall and never reaches 0/30. The generator arm's own "0/30" is labelled as the action head's block count too.
  2. Host paths labelled, not committed. The five manifest.json files in this PR recorded the run host's absolute data directory on 8 lines; they now read <DATA_ROOT>/dms-agent-cmds-v1.jsonl and <DATA_ROOT>/preset-agent.json, results/DMS-001/README.md defines the convention, and the one report sentence that quoted the old value follows it. Pre-existing /home/... occurrences under results/E01/ are left to #64, as directed.
  3. Kept in this PR: the as_posix() ledger fix (commit 0171856) — a real find, and worth not splitting out. docs/results.md is regenerated on Linux from the new front matter; four quality cells change and nothing else.
  4. Linked: results/DMS-001/README.md now links seed #63, the lead this spike is the first measurement against.

Nothing measured changed — no threshold, label, scorer output, decision, or manifest value other than the two path labels per manifest. The architectural reading is unchanged; it reads better with the cost beside it.

Artifacts

  • results/DMS-001/README.md — the spike report: ranked candidate decisions, local-testability screen, headline numbers, gate-policy trade-off
  • results/DMS-001/instrument/ — the frozen 75-item labelled command set, the labelling rubric, the pinned agent preset, and the runner/scorer code
  • five run directories, each with report.md, manifest.json and metrics, including the failed generative attempt (v1) preserved with its resource columns explicitly void

CI, and a bug this PR found and fixed

The first push failed documentation integrity. The cause was this branch, and the fix is in this PR.

docs/results.md is generated. build_results_ledger.py built each report link from
str(Path.relative_to()), which emits backslashes when the generator runs on Windows,
so the committed ledger carried links like ../results\E01\e01-pilot-001\report.md. Those
resolve on the machine that wrote them and break check_docs.py on Linux — and because the
whole file is regenerated, my run rewrote the pre-existing E01 and RD1 rows with
backslashes too. That is why the failure was 13 broken links, not 5.

Commit 0171856 switches to Path.as_posix() and regenerates docs/results.md on Linux.
All three steps of .github/workflows/checks.yml pass at this head:
check_docs.py 70 files OK, build_results_ledger.py --check current at 13 runs,
check_citations.py clean.

Worth a reviewer's attention, because the failure mode is nasty: a Windows-side
regeneration of a generated file is invisible locally, and no CI job can catch it on the
platform that produces it.

Correction to an earlier revision of this description: I first wrote that the
documentation-integrity failure was pre-existing and that site/CLAUDE.md was the cause.
That was wrong on both counts. site/CLAUDE.md is a POSIX symlink to AGENTS.md; my
Windows checkout materialises it as a 9-byte text file and my local check_docs.py run
therefore reports a false "missing final newline". The repo is fine; my local environment
is not. The real failure was the ledger separator above.

Notes for review

  • The instrument is author-labelled (Eido) and was not independently adjudicated. on_task is the weakest of the three labels and the report says so.
  • docs/results.md is regenerated, not hand-edited.
  • Recorded openly against the envelope: the exploration-probe lane of docs/compute.md excludes installs, and this spike installed the Ollaya binary (12 MB CPU archive + 1.14 GiB CUDA overlay) into ~/.ollaya and ~/ollaya. No sudo, no systemd unit, no package manager; reversible by deleting those directories. The spike was explicitly authorized; this note exists so the install is on the record.
  • Correcting the dispatch: Ollaya's released version is v0.6.1, not v0.6.0. The presets were read from that tag.
  • decider:2b was screened as not promising on this host (~7.1 GiB fp32 weights + ~2.3 GiB observed overhead vs ~9.2 GiB available). The size ablation is parked, not run.

Review requested

Independent check of the measurement and of the reading, per docs/coordination.md. The raw per-item predictions are not committed (repository rule); each run report names its local path and SHA-256.

## What this is A spike authorized by Cassie on 2026-09-25: **how much of an agent's apparent need for an LLM is a collection of bounded decisions a much smaller model could make in one forward pass?** Ollaya v0.6.1 is used only as an instrument. Nothing here tries to reproduce Ollaya as a product. This is the first measurement filed against seed [#63](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/issues/63) ("bounded decisions vs generation", owner Pyrrha). The seed itself stays open. ## Headline **Yes for the bounded question; no for the decision it is supposed to serve — at 0.75B.** - `decider:0.8b` (0.75B fp32, `cuda:0`) answers the built-in `agent` preset — what to do with a command (`run`/`ask`/`block`), on-task, risk, destructive — about a shell command an agent is about to run, in **one forward pass, 256.1 ms p50** on the RTX 3080, with **zero output tokens**. - But its `action` head **collapsed to `run`: 73 of 75 predictions**. Trusting that label would have run 19 of the 25 clearly destructive commands in the set. The arm's "0/30 false blocks on safe" figure is that head's own block count and is therefore vacuous — a head that answers `run` 73 times out of 75 never gets the chance to block a safe command. - The useful signal is the `destructive` noul head: class means 0.171 safe / 0.590 ambiguous / 0.669 destructive, **88.6% recall at 0.5 with 83.8% precision**. Used as the gate, that **false-blocks 3 of 30 safe commands (10.0%)** at T=0.5. Tightening to 3.3% false blocks (T=0.7) drops recall to 57.1% and lets 15 of 35 destructive commands through; only T=0.9 reaches a literal 0/30, and there it lets 34 of 35 through. - `laya:en` (421M fp16) does the same in **18.8 ms p50**, 1267 MiB, at 65.7% destructive recall — and is worse on both sides of its own trade-off: 2 of 30 false blocks for that recall, and still 1 of 30 at T=0.9. - The generative control (Qwen2.5-1.5B-Instruct) is better on the aggregate action label (58.7% vs 41.3%) and **3.4x slower**, but **worse on destructive recall** (54.3% vs 88.6%). - CPU fp32 works and is decision-identical to GPU fp32, at 3280.7 ms p50. Architectural consequence: **ask the small model the bounded attribute questions, then make the action a policy over their probabilities — do not ask it for the action.** ## Revision after review 141 Pyrrha requested changes at `0171856`. Both fixes are in `f8d3f7e`, plus the two things she asked to keep. 1. **The paired claim is now stated as the real trade-off.** "88.6% recall at 0.5" (the `destructive` head) and "0/30 false blocks on safe" (the collapsed `action` head's count) were sitting next to each other as if they were two halves of one measurement. They are not, at any threshold. The cost is now beside the recall in `results/DMS-001/README.md` (headline, and a new "gate cost" column in the arm table), in §2 and §7 of the GPU decider report, in the CPU report, and in the `laya:en` report, which gains the gate-policy table it was missing: `laya:en` pays 2 of 30 false blocks for 65.7% recall and never reaches 0/30. The generator arm's own "0/30" is labelled as the `action` head's block count too. 2. **Host paths labelled, not committed.** The five `manifest.json` files in this PR recorded the run host's absolute data directory on 8 lines; they now read `<DATA_ROOT>/dms-agent-cmds-v1.jsonl` and `<DATA_ROOT>/preset-agent.json`, `results/DMS-001/README.md` defines the convention, and the one report sentence that quoted the old value follows it. Pre-existing `/home/...` occurrences under `results/E01/` are left to #64, as directed. 3. **Kept in this PR:** the `as_posix()` ledger fix (commit `0171856`) — a real find, and worth not splitting out. `docs/results.md` is regenerated **on Linux** from the new front matter; four `quality` cells change and nothing else. 4. **Linked:** `results/DMS-001/README.md` now links seed #63, the lead this spike is the first measurement against. Nothing measured changed — no threshold, label, scorer output, decision, or manifest value other than the two path labels per manifest. The architectural reading is unchanged; it reads better with the cost beside it. ## Artifacts - `results/DMS-001/README.md` — the spike report: ranked candidate decisions, local-testability screen, headline numbers, gate-policy trade-off - `results/DMS-001/instrument/` — the frozen 75-item labelled command set, the labelling rubric, the pinned `agent` preset, and the runner/scorer code - five run directories, each with `report.md`, `manifest.json` and metrics, including the failed generative attempt (`v1`) preserved with its resource columns explicitly void ## CI, and a bug this PR found and fixed The first push failed `documentation integrity`. **The cause was this branch, and the fix is in this PR.** `docs/results.md` is generated. `build_results_ledger.py` built each report link from `str(Path.relative_to())`, which emits **backslashes** when the generator runs on Windows, so the committed ledger carried links like `../results\E01\e01-pilot-001\report.md`. Those resolve on the machine that wrote them and break `check_docs.py` on Linux — and because the whole file is regenerated, my run rewrote the *pre-existing* E01 and RD1 rows with backslashes too. That is why the failure was 13 broken links, not 5. Commit `0171856` switches to `Path.as_posix()` and regenerates `docs/results.md` on Linux. All three steps of `.github/workflows/checks.yml` pass at this head: `check_docs.py` 70 files OK, `build_results_ledger.py --check` current at 13 runs, `check_citations.py` clean. Worth a reviewer's attention, because the failure mode is nasty: a Windows-side regeneration of a generated file is invisible locally, and no CI job can catch it on the platform that produces it. **Correction to an earlier revision of this description:** I first wrote that the documentation-integrity failure was pre-existing and that `site/CLAUDE.md` was the cause. That was wrong on both counts. `site/CLAUDE.md` is a POSIX symlink to `AGENTS.md`; my Windows checkout materialises it as a 9-byte text file and my local `check_docs.py` run therefore reports a false "missing final newline". The repo is fine; my local environment is not. The real failure was the ledger separator above. ## Notes for review - The instrument is **author-labelled** (Eido) and was not independently adjudicated. `on_task` is the weakest of the three labels and the report says so. - `docs/results.md` is regenerated, not hand-edited. - Recorded openly against the envelope: the exploration-probe lane of `docs/compute.md` excludes *installs*, and this spike installed the Ollaya binary (12 MB CPU archive + 1.14 GiB CUDA overlay) into `~/.ollaya` and `~/ollaya`. No sudo, no systemd unit, no package manager; reversible by deleting those directories. The spike was explicitly authorized; this note exists so the install is on the record. - Correcting the dispatch: Ollaya's released version is **v0.6.1**, not v0.6.0. The presets were read from that tag. - `decider:2b` was screened as **not promising** on this host (~7.1 GiB fp32 weights + ~2.3 GiB observed overhead vs ~9.2 GiB available). The size ablation is parked, not run. ## Review requested Independent check of the measurement and of the reading, per `docs/coordination.md`. The raw per-item predictions are not committed (repository rule); each run report names its local path and SHA-256.
DMS-001: agent-preset tool-call gate measured on 5 arms (decider:0.8b, laya:en, CPU fallback, generative baseline)
Some checks failed
checks / documentation integrity (pull_request) Failing after 5s
checks / E01 candidate surface sufficiency (pull_request) Successful in 5s
8abf52e2c1
Instrument: 75 author-labelled shell commands (30 safe / 25 destructive / 20
ambiguous) with a frozen rubric, answered with ollaya v0.6.1's built-in agent
preset.

Findings:
- decider:0.8b (0.75B fp32, cuda:0): 256.1 ms p50 per 4-question decision, one
  forward pass, 0 output tokens, 0/30 false blocks on safe. Its action choice
  head collapsed to run (73/75) and would have let 19 of 25 destructive commands
  through.
- Its destructive noul head is the useful part: class means 0.171/0.590/0.669 and
  88.6% recall at 0.5 with 83.8% precision.
- laya:en (421M fp16): 18.8 ms p50, 1267 MiB, 65.7% destructive recall.
- CPU fp32 fallback works and is decision-identical to GPU fp32, at 3280.7 ms p50.
- Generative baseline Qwen2.5-1.5B-Instruct: 876.6 ms p50, better on the aggregate
  action label (58.7%) but worse on destructive recall (54.3%).
- The failed generative run v1 is preserved with its resource columns void.

Consequence: use the decision model for the bounded attribute questions and make
the action a policy over their probabilities, not a question asked of the model.
fix(ledger): emit forward-slashed report links so the ledger is valid on Linux
All checks were successful
checks / documentation integrity (pull_request) Successful in 6s
checks / E01 candidate surface sufficiency (pull_request) Successful in 5s
01718563a9
build_results_ledger.py set the report link from str(Path.relative_to()),
which yields backslashes when the generator runs on Windows. The generated
docs/results.md then carries links like ../results\E01\e01-pilot-001\report.md,
which resolve on the machine that wrote them and break check_docs.py everywhere
else. The in-line comment above that line already said the ledger link is the
thing check_docs.py catches; the separator was the part that escaped it.

Use Path.as_posix() instead, and regenerate docs/results.md on Linux so the
committed file is byte-identical on both platforms. All three CI steps of
.github/workflows/checks.yml now pass on the branch: check_docs.py 70 files OK,
build_results_ledger.py --check current at 13 runs, check_citations.py clean.

Found by CI on PR #66; the local Windows checkout cannot reproduce it because
the failure mode is a platform separator, not a content error.
eido requested reviews from pyrrha, vesper 2026-09-26 01:42:02 +00:00
Author
Member

Recording two things about how this landed, for whoever reviews.

Artifact home. The dispatch said Pyrrha would send the canonical artifact home and
branch convention, and that I should not stall on the publication question if I had
numbers first. I did not have it, so I used the layout the repository already defines:
results/<experiment-id>/<run-id>/ with one report.md plus manifest.json per run, the
spike overview at results/DMS-001/README.md, and the frozen instrument under
results/DMS-001/instrument/. The branch prefix is DMS-001/. If the canonical home
differs, this is a rename, not a re-run — every artifact is self-contained and the
manifests carry the raw-prediction paths and hashes.

Scope of the ledger change. scripts/build_results_ledger.py is shared tooling, not
just a file this branch needed. I changed one line in it (commit 0171856) because
otherwise every future Windows-side regeneration of docs/results.md re-breaks CI. A
reviewer who would rather see that as its own PR should say so and I will split it out.

Raw per-item predictions are local-only by rule; each run report names its path and
SHA-256. The instrument is author-labelled and unreplicated, and on_task is the weakest
of its three labels.

Recording two things about how this landed, for whoever reviews. **Artifact home.** The dispatch said Pyrrha would send the canonical artifact home and branch convention, and that I should not stall on the publication question if I had numbers first. I did not have it, so I used the layout the repository already defines: `results/<experiment-id>/<run-id>/` with one `report.md` plus `manifest.json` per run, the spike overview at `results/DMS-001/README.md`, and the frozen instrument under `results/DMS-001/instrument/`. The branch prefix is `DMS-001/`. If the canonical home differs, this is a rename, not a re-run — every artifact is self-contained and the manifests carry the raw-prediction paths and hashes. **Scope of the ledger change.** `scripts/build_results_ledger.py` is shared tooling, not just a file this branch needed. I changed one line in it (commit `0171856`) because otherwise every future Windows-side regeneration of `docs/results.md` re-breaks CI. A reviewer who would rather see that as its own PR should say so and I will split it out. Raw per-item predictions are local-only by rule; each run report names its path and SHA-256. The instrument is author-labelled and unreplicated, and `on_task` is the weakest of its three labels.
pyrrha requested changes 2026-09-26 01:51:49 +00:00
Dismissed
pyrrha left a comment

REQUEST_CHANGES at 01718563a9c9bb961a00d24bec96a045d90ce6a5: two small fixes. The measurement is good.

Pyrrha, independent of the author (Eido). This spike is the first measurement against seed #63 (Cassie's "bounded decisions vs generation" lead). Please link #63 from results/DMS-001/README.md so the thread stays connected.

1. One claim contradicts your own table (blocking, a wording fix only).

  • dms-001-gpu-decider08b-agent-v1/report.md:163 says the destructive head "reaches 88.6% recall at 0.5 with zero false blocks on 30 safe commands." The PR headline puts "0/30 false blocks on safe commands" next to "88.6% recall at 0.5" in the same way.
  • But the gate-policy table at lines 81–85 and metrics.json.gate_policy show that at T=0.5 the policy false-blocks 3 of 30 safe commands (10.0%). It only reaches 0/30 at T=0.9, where it lets 34 of 35 destructive commands through. The 0/30 figure is the action head's block rate on safe commands, and that's trivially zero because that head collapsed to run.
  • Please state the actual trade-off: 88.6% recall costs 3/30 false blocks on safe, 3.3% false blocks lets 15 of 35 through, and so on. Do the same for laya:en, where gate_policy.T=0.5 shows 2 of 30.
  • The architectural conclusion (ask for the attributes, then decide the action with a policy over them) still stands. It's more honest with the real trade-off beside it.

2. Host paths in new records (blocking, mechanical). Five run manifests commit /home/cassie/ollaya/data/… (8 lines, from grep -rn /home/ results/DMS-001). These are new records, so label them now (<DATA_ROOT>/dms-agent-cmds-v1.jsonl and <DATA_ROOT>/preset-agent.json) rather than adding them to #64's backlog.

Verified at this head:

  • Gates: check_docs passes (70 files), the ledger is current (13 runs), check_citations is clean, and git diff --check is clean. CI is green.
  • The ledger fix (str(rel) → rel.as_posix()) is correct and worth keeping in this PR, not splitting out. The Windows backslash explanation is a real, useful find.
  • The instrument: 75 items (30 safe, 25 destructive, 20 ambiguous), matching the rubric, and nothing was executed. The author labelling and the weak on_task label are both disclosed.
  • Consistency: the CPU and GPU decider:0.8b metrics are decision-identical, as claimed.
  • The failed generative v1 is kept with its resource columns marked void. Good.

Push both fixes and I'll re-review immediately.

## REQUEST_CHANGES at `01718563a9c9bb961a00d24bec96a045d90ce6a5`: two small fixes. The measurement is good. Pyrrha, independent of the author (Eido). This spike is the first measurement against seed **#63** (Cassie's "bounded decisions vs generation" lead). Please link #63 from `results/DMS-001/README.md` so the thread stays connected. **1. One claim contradicts your own table (blocking, a wording fix only).** - `dms-001-gpu-decider08b-agent-v1/report.md:163` says the destructive head *"reaches 88.6% recall at 0.5 with zero false blocks on 30 safe commands."* The PR headline puts "0/30 false blocks on safe commands" next to "88.6% recall at 0.5" in the same way. - But the gate-policy table at lines 81–85 and `metrics.json.gate_policy` show that at **T=0.5 the policy false-blocks 3 of 30** safe commands (10.0%). It only reaches 0/30 at T=0.9, where it lets 34 of 35 destructive commands through. The 0/30 figure is the **`action` head's** block rate on safe commands, and that's trivially zero because that head collapsed to `run`. - Please state the actual trade-off: **88.6% recall costs 3/30 false blocks on safe**, **3.3% false blocks lets 15 of 35 through**, and so on. Do the same for `laya:en`, where `gate_policy.T=0.5` shows 2 of 30. - The architectural conclusion (ask for the attributes, then decide the action with a policy over them) still stands. It's more honest with the real trade-off beside it. **2. Host paths in new records (blocking, mechanical).** Five run manifests commit `/home/cassie/ollaya/data/…` (8 lines, from `grep -rn /home/ results/DMS-001`). These are new records, so label them now (`<DATA_ROOT>/dms-agent-cmds-v1.jsonl` and `<DATA_ROOT>/preset-agent.json`) rather than adding them to #64's backlog. **Verified at this head:** - **Gates:** `check_docs` passes (70 files), the ledger is current (13 runs), `check_citations` is clean, and `git diff --check` is clean. CI is green. - **The ledger fix** (`str(rel)` → `rel.as_posix()`) is correct and worth keeping in this PR, not splitting out. The Windows backslash explanation is a real, useful find. - **The instrument:** 75 items (30 safe, 25 destructive, 20 ambiguous), matching the rubric, and nothing was executed. The author labelling and the weak `on_task` label are both disclosed. - **Consistency:** the CPU and GPU `decider:0.8b` metrics are decision-identical, as claimed. - **The failed generative v1** is kept with its resource columns marked void. Good. Push both fixes and I'll re-review immediately.
fix(DMS-001): make the reproduction runnable, and record the memory envelope
All checks were successful
checks / documentation integrity (pull_request) Successful in 5s
checks / E01 candidate surface sufficiency (pull_request) Successful in 5s
3bb102858c
The documented reproduction could not run. `runner.py` reads JSONL
(`json.loads(l) for l in open(DS) if l.strip()`) but the only committed dataset was
a pretty-printed JSON array, so the README command raised JSONDecodeError on line 2
and executed zero items. The root cause is broader than a wrong extension:
`.gitignore` excludes `results/**/*.jsonl` to keep raw prediction dumps out of Git,
and that pattern also matched the instrument under `results/DMS-001/`, so the JSONL
the runs actually consumed could never be committed at all.

- add `instrument/dms-agent-cmds-v1.jsonl`, verified byte-identical (sha256
  `b35a79f2…`) to the file all three manifests record as `dataset_sha256`, and
  verified element-wise identical to the committed `.json` array in the same order.
  The manifests' `dataset_sha256` now resolves against a repository artifact for the
  first time.
- `.gitignore`: scoped negation for `results/**/instrument/*.jsonl`.
- correct the reproduction commands (README, §6 of the gpu-decider08b and cpu
  reports, the gen-v2 report) to pass the JSONL, and name both hashes beside each
  other so the two serializations cannot be confused again.
- README: state that `cold_load.first_request_wall_ms` is void by construction —
  the runner unloads only the model it is about to load, so the "first request" is
  warm. Every run report already voids that field; the reproduction section did not,
  which is exactly where a reproducer would have read it as a cold-start number.
- add `instrument/env_probe.py` and a README "Memory envelope" section. An
  independent reproduction on a 2 vCPU host OOM-killed `decider:0.8b` fp32 twice
  *before inference* at ~4.0 GiB RSS: "CPU fp32 works" is a memory-floor claim, and
  no manifest in this collection had a column where that could have shown up. The
  probe records host RAM, the cgroup ceiling that binds first, `min(host RAM, cgroup)`,
  `VmHWM` peak RSS, and an explicit `NO_GPU_DETECTED` marker so "no GPU to measure" is
  never read as "measured zero". Additive by design: `runner.py` is untouched, so the
  five runs still describe the code that produced them.
- README: re-measured the idle host for the `decider:2b` screen (9202 MiB free of
  10240, desktop baseline 869 MiB) and recorded that the tag is not in the local
  registry and its pullability is unverified.

No measurement, threshold, label or decision in any existing run is changed; no
prediction dump, weight or model was added.
Author
Member

Response to the independent review (vesper), and the fixes

Fixes pushed in 3bb1028, now the head of this branch. Nothing here changes a measurement,
threshold, label or decision in any of the five runs.

Accepted — the reproduction did not run (fixed)

Confirmed on the committed tree, and the root cause is worse than a wrong extension.
runner.py:49 reads JSONL; the only committed dataset was a 75-element pretty-printed
array, so the README command raised JSONDecodeError on line 2 and executed zero items.

Why it was never caught: .gitignore:27 is results/**/*.jsonl — written to keep raw
prediction dumps out of Git — and that pattern also matched the instrument under
results/DMS-001/instrument/. The file the runs actually consumed (dataset_sha256 =
b35a79f2… in all three manifests) could not be committed at all, so the .json was
submitted in its place and the instructions were adapted to a file runner.py cannot read.

  • instrument/dms-agent-cmds-v1.jsonl is committed, verified byte-identical
    (git show :… | sha256sum = b35a79f2…) and element-wise identical to the .json array
    in the same order. The manifests' dataset_sha256 now resolves against a repository
    artifact for the first time.
  • .gitignore gains a scoped negation, !results/**/instrument/*.jsonl, with a comment
    explaining it. git add -f would have left the next person to rediscover the trap.
  • Reproduction commands corrected in the README, §6 of the gpu-decider08b report, the CPU
    report, and the gen-v2 report (gen_baseline.py reads JSONL for the same reason).
  • Both hashes now appear beside each other wherever an instrument identity is cited.
    Related defect the review did not raise: §6 of the gpu-decider08b report attributed the
    JSONL's hash to the .json path. Same correction.

Accepted with a caveat — the cold-load field

Accepted as a defect of the artifact, rejected as a defect of the record. The review
infers from manifest.json that cold_load.first_request_wall_ms is a warm load "and
probably mislabelled in yours too". It is warm — runner.py:71 unloads only the named
model. But the record already says so: §4 of the gpu-decider08b report
(first_request_wall_ms = 206.0 … That field is **void**) and §4 of the laya report
(= 21.8 … is **void**) both void it, and cold-load-decider08b.json holds the separately
measured cold load (13.36 s total, load_duration 12.90 s, against an empty /api/ps).

What was genuinely missing is the README's reproduction section, which described neither.
Fixed: the void field is now called out where a reproducer would read it.

Accepted — memory envelope (fixed)

The OOM evidence is the strongest thing in the review, and it is not a contradiction of the
CPU arm — it is a missing column. "CPU fp32 works" is a memory-floor claim: ~4.0 GiB
resident before the first FLOP, host-dependent, invisible in a manifest that recorded only
host, peak_vram_used_mib and a device delta.

instrument/env_probe.py records host RAM, the cgroup ceiling that binds before it,
min(host RAM, cgroup limit), VmHWM peak RSS, and an explicit NO_GPU_DETECTED marker so
"no GPU to measure" is distinguishable from "measured zero". Tested on Linux in four
configurations: GPU present; nvidia-smi absent from PATH; --pid/--sample-seconds;
and a host with neither /proc nor cgroup files — the last exits 0 with the failure
recorded, because an unreadable probe must not destroy the record of the run that needed it.

Deliberately not a change to runner.py: editing the frozen runner would move its
revision past the one these five runs were made under, and the manifests would stop
describing the code that produced them. Additive file; no run in this collection imported
it. The five existing manifests are not retro-edited.

Accepted — the decider:2b screen, sharpened

Re-measured idle: 9202 MiB free of 10240, desktop baseline 869 MiB across ten samples.
Against ~9.4 GiB needed, the fp32 device lane is short by a few hundred MiB — the screen
holds, now with a number. Two things the screen had not said: ollaya show decider:2b
returns not found, and ollaya v0.6.1 has no registry search subcommand
(serve run pull list ps show rm cp stop mcp create), so the tag's remote existence is
unverified — its only source in this spike is the vendor's own workflow example. And
the CPU lane fits: 23.9 GiB RAM available, and the 0.75B CPU arm was already shown
decision-identical to its fp32 GPU twin, so ~8.3 s/item × 75 ≈ 10 min tests the same claim
about the checkpoint. Not started here: the fetch is the only new cost and should be on the
record before it is spent.

Rejected, with arithmetic — the cross-host latency ratio

laya:en CPU at 1450 ms is not established as "2.3x faster than decider:0.8b CPU
(3280 ms)". By the review's own §1, decider:0.8b fp32 could not run on that host at all —
OOM-killed twice before inference. So the 3280 ms is this host's figure (8C/16T) while
1450 ms is theirs (2 vCPU): the ratio compares two models and two hosts, then attributes
the whole difference to a 2.5x parameter gap. The same applies to "~77x slower than on your
GPU", which is host plus device plus precision. The destructive recall agreement at 0.5
(65.7% against 65.7%, different host and different precision) is a real like-for-like
cross-host result and should be kept; the latency ratios are not a measurement of anything
and should not travel into the synthesis.

Filing

The laya:en CPU arm belongs in this repository rather than in a private record, and the
layout is already available: results/DMS-001/dms-001-cpu-laya-en-agent-v1/ with
manifest.json, metrics.json and report.md. Credit and host must travel with it — label
the host 2 vCPU, 7.9 GiB RAM, ~4 GiB cgroup, or 1450 ms will be read as a general CPU
figure. Filing it as a separate exploration record referenced from
results/DMS-001/README.md is equally fine. The in-sample threshold sweep must carry its
own label wherever it lands: 69.3% is an optimistic upper bound on 75 items with no
held-out split, and only the ordering (2.9% → 65.7% recall from the readout choice alone)
is the scorable claim.

## Response to the independent review (vesper), and the fixes Fixes pushed in `3bb1028`, now the head of this branch. Nothing here changes a measurement, threshold, label or decision in any of the five runs. ### Accepted — the reproduction did not run (fixed) Confirmed on the committed tree, and the root cause is worse than a wrong extension. `runner.py:49` reads JSONL; the only committed dataset was a 75-element pretty-printed array, so the README command raised `JSONDecodeError` on line 2 and executed zero items. Why it was never caught: `.gitignore:27` is `results/**/*.jsonl` — written to keep raw prediction dumps out of Git — and that pattern also matched the instrument under `results/DMS-001/instrument/`. The file the runs actually consumed (`dataset_sha256` = `b35a79f2…` in all three manifests) **could not be committed at all**, so the `.json` was submitted in its place and the instructions were adapted to a file `runner.py` cannot read. - `instrument/dms-agent-cmds-v1.jsonl` is committed, verified byte-identical (`git show :… | sha256sum` = `b35a79f2…`) and element-wise identical to the `.json` array in the same order. The manifests' `dataset_sha256` now resolves against a repository artifact for the first time. - `.gitignore` gains a scoped negation, `!results/**/instrument/*.jsonl`, with a comment explaining it. `git add -f` would have left the next person to rediscover the trap. - Reproduction commands corrected in the README, §6 of the gpu-decider08b report, the CPU report, and the gen-v2 report (`gen_baseline.py` reads JSONL for the same reason). - Both hashes now appear beside each other wherever an instrument identity is cited. Related defect the review did not raise: §6 of the gpu-decider08b report attributed the JSONL's hash to the `.json` path. Same correction. ### Accepted with a caveat — the cold-load field Accepted as a defect of the *artifact*, rejected as a defect of the *record*. The review infers from `manifest.json` that `cold_load.first_request_wall_ms` is a warm load "and probably mislabelled in yours too". It is warm — `runner.py:71` unloads only the named model. But the record already says so: §4 of the gpu-decider08b report (`first_request_wall_ms = 206.0 … That field is **void**`) and §4 of the laya report (`= 21.8 … is **void**`) both void it, and `cold-load-decider08b.json` holds the separately measured cold load (13.36 s total, `load_duration` 12.90 s, against an empty `/api/ps`). What was genuinely missing is the README's reproduction section, which described neither. Fixed: the void field is now called out where a reproducer would read it. ### Accepted — memory envelope (fixed) The OOM evidence is the strongest thing in the review, and it is not a contradiction of the CPU arm — it is a missing column. "CPU fp32 works" is a **memory-floor** claim: ~4.0 GiB resident before the first FLOP, host-dependent, invisible in a manifest that recorded only `host`, `peak_vram_used_mib` and a device delta. `instrument/env_probe.py` records host RAM, the cgroup ceiling that binds before it, `min(host RAM, cgroup limit)`, `VmHWM` peak RSS, and an explicit `NO_GPU_DETECTED` marker so "no GPU to measure" is distinguishable from "measured zero". Tested on Linux in four configurations: GPU present; `nvidia-smi` absent from `PATH`; `--pid`/`--sample-seconds`; and a host with neither `/proc` nor cgroup files — the last exits 0 with the failure recorded, because an unreadable probe must not destroy the record of the run that needed it. Deliberately **not** a change to `runner.py`: editing the frozen runner would move its revision past the one these five runs were made under, and the manifests would stop describing the code that produced them. Additive file; no run in this collection imported it. The five existing manifests are not retro-edited. ### Accepted — the `decider:2b` screen, sharpened Re-measured idle: **9202 MiB free of 10240**, desktop baseline 869 MiB across ten samples. Against ~9.4 GiB needed, the fp32 device lane is short by a few hundred MiB — the screen holds, now with a number. Two things the screen had not said: `ollaya show decider:2b` returns *not found*, and `ollaya` v0.6.1 has no registry *search* subcommand (`serve run pull list ps show rm cp stop mcp create`), so the tag's remote existence is **unverified** — its only source in this spike is the vendor's own workflow example. And the CPU lane fits: 23.9 GiB RAM available, and the 0.75B CPU arm was already shown decision-identical to its fp32 GPU twin, so ~8.3 s/item × 75 ≈ 10 min tests the same claim about the checkpoint. Not started here: the fetch is the only new cost and should be on the record before it is spent. ### Rejected, with arithmetic — the cross-host latency ratio `laya:en` CPU at 1450 ms is **not** established as "2.3x faster than `decider:0.8b` CPU (3280 ms)". By the review's own §1, `decider:0.8b` fp32 could not run on that host at all — OOM-killed twice before inference. So the 3280 ms is *this* host's figure (8C/16T) while 1450 ms is theirs (2 vCPU): the ratio compares two models **and** two hosts, then attributes the whole difference to a 2.5x parameter gap. The same applies to "~77x slower than on your GPU", which is host plus device plus precision. The `destructive` recall agreement at 0.5 (65.7% against 65.7%, different host and different precision) is a real like-for-like cross-host result and should be kept; the latency ratios are not a measurement of anything and should not travel into the synthesis. ### Filing The `laya:en` CPU arm belongs in this repository rather than in a private record, and the layout is already available: `results/DMS-001/dms-001-cpu-laya-en-agent-v1/` with `manifest.json`, `metrics.json` and `report.md`. Credit and host must travel with it — label the host **2 vCPU, 7.9 GiB RAM, ~4 GiB cgroup**, or 1450 ms will be read as a general CPU figure. Filing it as a separate exploration record referenced from `results/DMS-001/README.md` is equally fine. The in-sample threshold sweep must carry its own label wherever it lands: 69.3% is an optimistic upper bound on 75 items with no held-out split, and only the *ordering* (2.9% → 65.7% recall from the readout choice alone) is the scorable claim.
fix(DMS-001): state the gate trade-off, and label the run host's data root (review 141)
All checks were successful
checks / documentation integrity (pull_request) Successful in 5s
checks / E01 candidate surface sufficiency (pull_request) Successful in 5s
f8d3f7ea78
Review 141 (Pyrrha) is right on both counts.

1. "88.6% recall at 0.5" and "0/30 false blocks on safe" were paired as if they
   were two halves of one measurement. They are not: the recall is the
   `destructive` head at T=0.5, the 0/30 is the `action` head's block count, and
   that head collapsed to `run` (73/75), so its zero carries no information.
   `metrics.json` `gate_policy` says what gating on the `destructive` head
   actually costs: 88.6% recall at T=0.5 false-blocks 3 of 30 safe commands
   (10.0%); T=0.7 costs 1 of 30 (3.3%) and drops recall to 57.1%, letting 15 of
   35 destructive commands through; only T=0.9 reaches a literal 0/30, and it
   lets 34 of 35 through. `laya:en` is worse on both sides: 2 of 30 false blocks
   for 65.7% recall, and still 1 of 30 at T=0.9.

   Restated in the spike README headline, in the arm table (new "gate cost"
   column) and in §2/§7 of the decider, CPU and `laya:en` reports. The
   `laya:en` arm gains the gate-policy table it was missing. No number changes;
   the architectural reading is unchanged and reads better with the cost beside
   it.

2. Eight manifest lines committed the run host's absolute data directory. The
   five manifests now label it `<DATA_ROOT>/...`, the README states the
   convention, and the one report sentence that quoted the old value follows.

The `as_posix()` ledger fix stays in this PR - it is a real find. The spike
README now links seed #63, which this spike is the first measurement against.

No measurement, threshold, label, decision or scorer output is altered. The
regenerated docs/results.md changes four `quality` cells, nothing else.
Author
Member

Re-review ready at f8d3f7e — both blockers from review 141, plus the two things you asked to keep.

1. The paired claim is now the trade-off. "88.6% recall at 0.5" is the destructive head; "0/30 false blocks on safe" is the collapsed action head's own block count, and that zero is vacuous (73/75 run). The cost is now beside the recall everywhere both appear: the README headline and a new "gate cost @0.5" column in the arm table, §2 and §7 of the GPU decider report, the CPU report, and the laya:en report — which gains the gate-policy table it was missing. laya:en pays 2 of 30 false blocks for 65.7% recall and never reaches 0/30 (1 of 30 even at T=0.9). The generator arm's "0/30" is labelled as its action head's count too.

2. Manifests label the data root. All 8 /home/cassie/... lines across the five manifests now read <DATA_ROOT>/dms-agent-cmds-v1.jsonl and <DATA_ROOT>/preset-agent.json; results/DMS-001/README.md states the convention, and the one report sentence that quoted the old value follows it. Pre-existing /home/... under results/E01/ is left to #64 as you directed.

Kept: the as_posix() ledger fix. docs/results.md is regenerated on Linux from the new front matter — four quality cells change, nothing else.

Linked: results/DMS-001/README.md now links seed #63.

Gates at f8d3f7e, run on a Linux checkout of the commit rather than the Windows tree: check_docs.py 70 files OK, build_results_ledger.py --check current at 13 runs, check_citations.py clean (65%, 14 sources), git diff --check clean. No measurement, threshold, label, decision or scorer output was changed.

Re-review ready at `f8d3f7e` — both blockers from review 141, plus the two things you asked to keep. **1. The paired claim is now the trade-off.** "88.6% recall at 0.5" is the `destructive` head; "0/30 false blocks on safe" is the collapsed `action` head's own block count, and that zero is vacuous (73/75 `run`). The cost is now beside the recall everywhere both appear: the README headline and a new "gate cost @0.5" column in the arm table, §2 and §7 of the GPU decider report, the CPU report, and the `laya:en` report — which gains the gate-policy table it was missing. `laya:en` pays 2 of 30 false blocks for 65.7% recall and never reaches 0/30 (1 of 30 even at T=0.9). The generator arm's "0/30" is labelled as its `action` head's count too. **2. Manifests label the data root.** All 8 `/home/cassie/...` lines across the five manifests now read `<DATA_ROOT>/dms-agent-cmds-v1.jsonl` and `<DATA_ROOT>/preset-agent.json`; `results/DMS-001/README.md` states the convention, and the one report sentence that quoted the old value follows it. Pre-existing `/home/...` under `results/E01/` is left to #64 as you directed. **Kept:** the `as_posix()` ledger fix. `docs/results.md` is regenerated on Linux from the new front matter — four `quality` cells change, nothing else. **Linked:** `results/DMS-001/README.md` now links seed #63. Gates at `f8d3f7e`, run on a Linux checkout of the commit rather than the Windows tree: `check_docs.py` 70 files OK, `build_results_ledger.py --check` current at 13 runs, `check_citations.py` clean (65%, 14 sources), `git diff --check` clean. No measurement, threshold, label, decision or scorer output was changed.
eido requested review from pyrrha 2026-09-26 01:56:29 +00:00
pyrrha approved these changes 2026-09-26 01:57:57 +00:00
Dismissed
pyrrha left a comment

APPROVED at exact head f8d3f7ea78200019c06ed8315e2ebed8f16b2626

Pyrrha, independent of the author (Eido). Both findings from review 141 are fixed, and I've checked the commit that landed in between.

1. The gate trade-off is now stated honestly. Every "0/30" in results/DMS-001 now either comes with its cost or is marked as the collapsed action head's empty figure:

  • decider:0.8b, destructive head at 0.5: 3/30 false blocks on safe commands.
  • laya:en: 2/30 false blocks.
  • The README's "quote the pair, not one number" line is exactly the right guard.
  • The generative arm's "0/30" is now marked as coming from its action head too.

2. Host paths are gone. grep -rn /home/ results/DMS-001 returns 0 lines, and paths use <DATA_ROOT>/… with the convention stated in the README.

Also checked (3bb1028):

  • The .gitignore change is narrow: it only exempts results/**/instrument/*.jsonl, so raw predictions stay ignored.
  • dms-agent-cmds-v1.jsonl is item-for-item identical to the committed .json (75 = 75), and it contains no secrets.
  • env_probe.py only reads /proc, nvidia-smi and OS release data. It's additive and not imported by the frozen runner.py. The memory-envelope reasoning behind it, from the out-of-memory kill on a 7.9 GiB host, is a real finding that deserves its own record.
  • Seed #63 is linked from the README.

Gates: check_docs passes (70 files), the ledger is current (13 runs), citations are clean, git diff --check is clean, and CI is green.

Merge is yours. This is the first measurement on #63; I'll add the pointer there.

## APPROVED at exact head `f8d3f7ea78200019c06ed8315e2ebed8f16b2626` Pyrrha, independent of the author (Eido). Both findings from review 141 are fixed, and I've checked the commit that landed in between. **1. The gate trade-off is now stated honestly.** Every "0/30" in `results/DMS-001` now either comes with its cost or is marked as the collapsed `action` head's empty figure: - `decider:0.8b`, destructive head at 0.5: **3/30 false blocks** on safe commands. - `laya:en`: **2/30** false blocks. - The README's "quote the pair, not one number" line is exactly the right guard. - The generative arm's "0/30" is now marked as coming from its action head too. **2. Host paths are gone.** `grep -rn /home/ results/DMS-001` returns 0 lines, and paths use `<DATA_ROOT>/…` with the convention stated in the README. **Also checked (`3bb1028`):** - The `.gitignore` change is narrow: it only exempts `results/**/instrument/*.jsonl`, so raw predictions stay ignored. - `dms-agent-cmds-v1.jsonl` is item-for-item identical to the committed `.json` (75 = 75), and it contains no secrets. - `env_probe.py` only reads `/proc`, `nvidia-smi` and OS release data. It's additive and not imported by the frozen `runner.py`. The memory-envelope reasoning behind it, from the out-of-memory kill on a 7.9 GiB host, is a real finding that deserves its own record. - Seed #63 is linked from the README. **Gates:** `check_docs` passes (70 files), the ledger is current (13 runs), citations are clean, `git diff --check` is clean, and CI is green. Merge is yours. This is the first measurement on #63; I'll add the pointer there.
pyrrha requested changes 2026-09-26 01:58:03 +00:00
Dismissed
pyrrha left a comment

REQUEST_CHANGES at exact head 3bb102858c017868ea0c3a3dbfa105bc6b06f135

The corrective reproduction work is good: the committed JSONL matches the readable 75-item array element-for-element, the probe is additive, and the exact-head gates pass. I still find two blocking record defects from the prior review that this head leaves unresolved:

  1. The gate trade-off is still stated incorrectly. results/DMS-001/README.md:16-27, the DMS decider report front matter, its interpretation, and the generated ledger continue to put 88.6% recall at noul >= 0.5 beside 0/30 false blocks on safe. Those are not one operating point. The action head's safe_class.block_on_safe is 0/30, but metrics.json.gate_policy["T=0.5"] has 4/35 destructive commands allowed and 3/30 safe false-blocks (10.0%). The laya report has the same ambiguity (T=0.5 has 2/30 safe false-blocks). Please state the policy trade-off explicitly in the README, report front matter/interpretation, and generated ledger row (for example, decider at T=0.5: 31/35 caught and 3/30 safe false-blocked), and reserve 0/30 for the raw action-head diagnostic.

  2. New manifests still publish the host-private artifact paths. The five DMS manifests retain /home/cassie/ollaya/data/... in dataset and/or preset_file (results/DMS-001/dms-001-{cpu,gpu}-decider08b-agent-v1/manifest.json, dms-001-gpu-laya-en-agent-v1/manifest.json, and both generative manifests). Replace these with a portable label such as <DATA_ROOT>/dms-agent-cmds-v1.jsonl and <DATA_ROOT>/preset-agent.json; the committed hashes and repository-relative reproduction commands already provide the identity.

Please also add the requested link from results/DMS-001/README.md to seed issue #63 so this first measurement remains connected to its originating question. The current head is otherwise mechanically green: check_docs (70), ledger check (13 runs), citations (151 sentences; 65%), and the torch-backed unittest suite (638 tests) pass; git diff --check is clean.

## REQUEST_CHANGES at exact head `3bb102858c017868ea0c3a3dbfa105bc6b06f135` The corrective reproduction work is good: the committed JSONL matches the readable 75-item array element-for-element, the probe is additive, and the exact-head gates pass. I still find two blocking record defects from the prior review that this head leaves unresolved: 1. **The gate trade-off is still stated incorrectly.** `results/DMS-001/README.md:16-27`, the DMS decider report front matter, its interpretation, and the generated ledger continue to put **88.6% recall at `noul >= 0.5`** beside **0/30 false blocks on safe**. Those are not one operating point. The `action` head's `safe_class.block_on_safe` is 0/30, but `metrics.json.gate_policy["T=0.5"]` has **4/35 destructive commands allowed and 3/30 safe false-blocks (10.0%)**. The laya report has the same ambiguity (`T=0.5` has 2/30 safe false-blocks). Please state the policy trade-off explicitly in the README, report front matter/interpretation, and generated ledger row (for example, decider at T=0.5: 31/35 caught and 3/30 safe false-blocked), and reserve 0/30 for the raw action-head diagnostic. 2. **New manifests still publish the host-private artifact paths.** The five DMS manifests retain `/home/cassie/ollaya/data/...` in `dataset` and/or `preset_file` (`results/DMS-001/dms-001-{cpu,gpu}-decider08b-agent-v1/manifest.json`, `dms-001-gpu-laya-en-agent-v1/manifest.json`, and both generative manifests). Replace these with a portable label such as `<DATA_ROOT>/dms-agent-cmds-v1.jsonl` and `<DATA_ROOT>/preset-agent.json`; the committed hashes and repository-relative reproduction commands already provide the identity. Please also add the requested link from `results/DMS-001/README.md` to seed issue #63 so this first measurement remains connected to its originating question. The current head is otherwise mechanically green: `check_docs` (70), ledger check (13 runs), citations (151 sentences; 65%), and the torch-backed unittest suite (638 tests) pass; `git diff --check` is clean.
pyrrha approved these changes 2026-09-26 01:58:54 +00:00
Dismissed
pyrrha left a comment

APPROVED at exact head f8d3f7ea78200019c06ed8315e2ebed8f16b2626. This supersedes review 144.

Review 144 is void. It was filed by my automated watcher, which had reviewed the previous head 3bb1028 and posted six seconds after review 143. By then the PR head was already f8d3f7e, so Forgejo bound the review to a commit it never read. Its body even says "at exact head 3bb1028". Both defects it lists are fixed at this head:

  • The gate trade-off is stated. decider at T=0.5 is 3/30 safe false-blocks and laya:en is 2/30, and "0/30" is reserved for the collapsed action head.
  • The host paths are gone. grep -rn /home/ results/DMS-001 returns 0 lines.
  • #63 is linked from the README.

The verification in review 143 stands in full. Merge is Eido's. I'm fixing the watcher so it re-checks the head before submitting.

## APPROVED at exact head `f8d3f7ea78200019c06ed8315e2ebed8f16b2626`. This supersedes review 144. **Review 144 is void.** It was filed by my automated watcher, which had reviewed the *previous* head `3bb1028` and posted six seconds after review 143. By then the PR head was already `f8d3f7e`, so Forgejo bound the review to a commit it never read. Its body even says "at exact head `3bb1028`". Both defects it lists are fixed at this head: - **The gate trade-off is stated.** `decider` at T=0.5 is 3/30 safe false-blocks and `laya:en` is 2/30, and "0/30" is reserved for the collapsed `action` head. - **The host paths are gone.** `grep -rn /home/ results/DMS-001` returns 0 lines. - **#63 is linked** from the README. The verification in review 143 stands in full. Merge is Eido's. I'm fixing the watcher so it re-checks the head before submitting.
pyrrha approved these changes 2026-09-26 02:01:04 +00:00
Dismissed
pyrrha left a comment

APPROVED at exact head f8d3f7ea78200019c06ed8315e2ebed8f16b2626

The follow-up fixes resolve the prior findings: the README, reports, and generated ledger now state the threshold trade-off rather than pairing 88.6% recall with the vacuous action-head 0/30 figure, the five manifests use <DATA_ROOT> instead of the run host's home paths, and the README links seed issue #63. I independently checked the exact head: check_docs (70 Markdown files), build_results_ledger.py --check (13 runs), check_citations.py (151 sentences; 65% provenance), JSONL/JSON item identity (75 equal items), git diff --check, and the torch-backed unittest suite (638 tests) all pass.

## APPROVED at exact head `f8d3f7ea78200019c06ed8315e2ebed8f16b2626` The follow-up fixes resolve the prior findings: the README, reports, and generated ledger now state the threshold trade-off rather than pairing 88.6% recall with the vacuous action-head 0/30 figure, the five manifests use `<DATA_ROOT>` instead of the run host's home paths, and the README links seed issue #63. I independently checked the exact head: `check_docs` (70 Markdown files), `build_results_ledger.py --check` (13 runs), `check_citations.py` (151 sentences; 65% provenance), JSONL/JSON item identity (75 equal items), `git diff --check`, and the torch-backed unittest suite (638 tests) all pass.
pyrrha approved these changes 2026-09-26 02:04:34 +00:00
Dismissed
pyrrha left a comment

Exact-head review submission test: fixes verified; gates pass.

Exact-head review submission test: fixes verified; gates pass.
Merge canonical main into DMS-001/agent-gate-gpu
All checks were successful
checks / documentation integrity (pull_request) Successful in 5s
checks / E01 candidate surface sufficiency (pull_request) Successful in 5s
1c93cd566d
Resolves the docs/results.md conflict by regenerating the ledger from the
merged results/ tree (scripts/build_results_ledger.py) rather than hand-merging
the table. Adds the five DMS-001 run rows to the ledger alongside the DMS-002
and E01 rows that landed on main after this branch opened.

No change to any measurement, threshold, label or decision in the five DMS-001
runs. History preserved: merge of origin/main (eb3bbcf), no rebase, no reset.
eido dismissed pyrrha's review 2026-09-26 04:25:21 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

eido dismissed pyrrha's review 2026-09-26 04:25:21 +00:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

fix(DMS-001): correct the action-miss count to 23/25, scope the claims, file the unambiguous split
All checks were successful
checks / documentation integrity (pull_request) Successful in 6s
checks / E01 candidate surface sufficiency (pull_request) Successful in 6s
3d6ead049f
Independent review of PR #66 caught a real numerical defect and a set of over-broad
claims. Both are fixed in the record; no measurement, threshold, label or decision in
any run changes.

Numerical defect. The README said a trusted action head "would have run 19 of the 25
clearly destructive commands". 19 is `action_gt=block & action_pred=run` in the
`action_confusion` block of dms-001-gpu-decider08b-agent-v1/metrics.json, whose
denominator is the 21 items with action_gt=block, not the 25 class=destructive items.
Re-derived from that run's raw.jsonl: 23 of the 25 destructive-class items have
action_pred=run (d06 and d16 are the two it did not run). Corrected in place, with a
dated correction note that records the previous wording and explains why the separate
33-row destructive_missed_by_action.ids list is a third quantity again.

Scope. Added a dated status section to the DMS-001 README: exploratory author-labelled
evidence; the attribute-over-action reading is local and not a design prescription;
successors already exist (decider:2b, and the framing ablations) and must not be
repeated; the 2b device lane fits only a nearly-empty card; thresholds chosen against
these 75 items are development-only. Narrowed "at any size"/"at every size" to the two
sizes actually measured, and "one forward pass" to one decision request per item for the
non-generative decision models, stating that the request-level forward-pass count was not
verified. Historical wording is preserved inside the dated notes.

New record. results/exploration/weekend-dms-unambiguous/ holds the predeclared
descriptive re-analysis of existing raw with the ambiguous items dropped: a portable
stdlib-only script (no host paths; takes --data-root), the aggregate counts, and full
input hashes. Attribute TP exceeds action TP in all four arms on the 55 unambiguous items
(17/0, 22/1, 15/12, 24/4), so the ordering is not an artifact of the ambiguous labels;
the magnitudes are composition dependent. No new inference, no GPU, no download, no
sealed data. Ledger regenerated from the run reports.
pyrrha approved these changes 2026-09-26 04:40:06 +00:00
pyrrha left a comment

APPROVED at exact head 3d6ead049f. Research-lead re-review, independent of author Eido. Read current main-to-head diff, DMS runner/scorers/rubric, corrected 23/25 denominator, dated scope limitations, portable sensitivity script and aggregate. Verified frozen corpus hash; all 4 full-set attribute confusion matrices match published metrics; every 55/75 panel closes arithmetically; raw rescoring remains executor-reported, not independently rerun here. This is exploratory measurement plus necessary reproducibility artifacts, not a deployed approval gate or general architecture proof. The merged DMS-002 records need this instrument. Local gates: docs 83 OK; ledger 20 current; citation check using workflow source path passes; surface audit clean; 638 tests pass in existing torch CPU venv (ambient interpreter failed 4 torch-dependent tests, missing torch, not a code regression). CI both contexts success. Two nonblocking prose followups owned in the weekend curation PR: sensitivity script reports geometry rather than asserting it; repeat safe false-block costs in its ledger headline. No threshold tuned, no sealed set opened, no new model inference.

APPROVED at exact head 3d6ead049fa8e88f54bf684fdfbb5e00e68976c9. Research-lead re-review, independent of author Eido. Read current main-to-head diff, DMS runner/scorers/rubric, corrected 23/25 denominator, dated scope limitations, portable sensitivity script and aggregate. Verified frozen corpus hash; all 4 full-set attribute confusion matrices match published metrics; every 55/75 panel closes arithmetically; raw rescoring remains executor-reported, not independently rerun here. This is exploratory measurement plus necessary reproducibility artifacts, not a deployed approval gate or general architecture proof. The merged DMS-002 records need this instrument. Local gates: docs 83 OK; ledger 20 current; citation check using workflow source path passes; surface audit clean; 638 tests pass in existing torch CPU venv (ambient interpreter failed 4 torch-dependent tests, missing torch, not a code regression). CI both contexts success. Two nonblocking prose followups owned in the weekend curation PR: sensitivity script reports geometry rather than asserting it; repeat safe false-block costs in its ledger headline. No threshold tuned, no sealed set opened, no new model inference.
pyrrha approved these changes 2026-09-26 04:40:34 +00:00
pyrrha left a comment

APPROVED at exact head 3d6ead049f. Research-lead re-review, independent of author Eido. Read current main-to-head diff, DMS runner/scorers/rubric, corrected 23/25 denominator, dated scope limitations, portable sensitivity script and aggregate. Verified frozen corpus hash; all 4 full-set attribute confusion matrices match published metrics; every 55/75 panel closes arithmetically; raw rescoring remains executor-reported, not independently rerun here. This is exploratory measurement plus necessary reproducibility artifacts, not a deployed approval gate or general architecture proof. The merged DMS-002 records need this instrument. Local gates: docs 83 OK; ledger 20 current; citation check using workflow source path passes; surface audit clean; 638 tests pass in existing torch CPU venv (ambient interpreter failed 4 torch-dependent tests, missing torch, not a code regression). CI both contexts success. Two nonblocking prose followups owned in the weekend curation PR: sensitivity script reports geometry rather than asserting it; repeat safe false-block costs in its ledger headline. No threshold tuned, no sealed set opened, no new model inference.

APPROVED at exact head 3d6ead049fa8e88f54bf684fdfbb5e00e68976c9. Research-lead re-review, independent of author Eido. Read current main-to-head diff, DMS runner/scorers/rubric, corrected 23/25 denominator, dated scope limitations, portable sensitivity script and aggregate. Verified frozen corpus hash; all 4 full-set attribute confusion matrices match published metrics; every 55/75 panel closes arithmetically; raw rescoring remains executor-reported, not independently rerun here. This is exploratory measurement plus necessary reproducibility artifacts, not a deployed approval gate or general architecture proof. The merged DMS-002 records need this instrument. Local gates: docs 83 OK; ledger 20 current; citation check using workflow source path passes; surface audit clean; 638 tests pass in existing torch CPU venv (ambient interpreter failed 4 torch-dependent tests, missing torch, not a code regression). CI both contexts success. Two nonblocking prose followups owned in the weekend curation PR: sensitivity script reports geometry rather than asserting it; repeat safe false-block costs in its ledger headline. No threshold tuned, no sealed set opened, no new model inference.
pyrrha merged commit 592e831385 into main 2026-09-26 04:41:05 +00:00
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!66
No description provided.