DMS-001: agent-preset tool-call gate measured on 5 arms (decider:0.8b, laya:en, CPU, generative baseline) #66
Loading…
Reference in a new issue
No description provided.
Delete branch "DMS-001/agent-gate-gpu"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this is
A spike authorized by Cassie on 2026-09-25: how much of an agent's apparent need for an LLM is a collection of bounded decisions a much smaller model could make in one forward pass?
Ollaya v0.6.1 is used only as an instrument. Nothing here tries to reproduce Ollaya as a product.
This is the first measurement filed against seed #63 ("bounded decisions vs generation", owner Pyrrha). The seed itself stays open.
Headline
Yes for the bounded question; no for the decision it is supposed to serve — at 0.75B.
decider:0.8b(0.75B fp32,cuda:0) answers the built-inagentpreset — what to do with a command (run/ask/block), on-task, risk, destructive — about a shell command an agent is about to run, in one forward pass, 256.1 ms p50 on the RTX 3080, with zero output tokens.actionhead collapsed torun: 73 of 75 predictions. Trusting that label would have run 19 of the 25 clearly destructive commands in the set. The arm's "0/30 false blocks on safe" figure is that head's own block count and is therefore vacuous — a head that answersrun73 times out of 75 never gets the chance to block a safe command.destructivenoul head: class means 0.171 safe / 0.590 ambiguous / 0.669 destructive, 88.6% recall at 0.5 with 83.8% precision. Used as the gate, that false-blocks 3 of 30 safe commands (10.0%) at T=0.5. Tightening to 3.3% false blocks (T=0.7) drops recall to 57.1% and lets 15 of 35 destructive commands through; only T=0.9 reaches a literal 0/30, and there it lets 34 of 35 through.laya:en(421M fp16) does the same in 18.8 ms p50, 1267 MiB, at 65.7% destructive recall — and is worse on both sides of its own trade-off: 2 of 30 false blocks for that recall, and still 1 of 30 at T=0.9.Architectural consequence: ask the small model the bounded attribute questions, then make the action a policy over their probabilities — do not ask it for the action.
Revision after review 141
Pyrrha requested changes at
0171856. Both fixes are inf8d3f7e, plus the two things she asked to keep.destructivehead) and "0/30 false blocks on safe" (the collapsedactionhead's count) were sitting next to each other as if they were two halves of one measurement. They are not, at any threshold. The cost is now beside the recall inresults/DMS-001/README.md(headline, and a new "gate cost" column in the arm table), in §2 and §7 of the GPU decider report, in the CPU report, and in thelaya:enreport, which gains the gate-policy table it was missing:laya:enpays 2 of 30 false blocks for 65.7% recall and never reaches 0/30. The generator arm's own "0/30" is labelled as theactionhead's block count too.manifest.jsonfiles in this PR recorded the run host's absolute data directory on 8 lines; they now read<DATA_ROOT>/dms-agent-cmds-v1.jsonland<DATA_ROOT>/preset-agent.json,results/DMS-001/README.mddefines the convention, and the one report sentence that quoted the old value follows it. Pre-existing/home/...occurrences underresults/E01/are left to #64, as directed.as_posix()ledger fix (commit0171856) — a real find, and worth not splitting out.docs/results.mdis regenerated on Linux from the new front matter; fourqualitycells change and nothing else.results/DMS-001/README.mdnow links seed #63, the lead this spike is the first measurement against.Nothing measured changed — no threshold, label, scorer output, decision, or manifest value other than the two path labels per manifest. The architectural reading is unchanged; it reads better with the cost beside it.
Artifacts
results/DMS-001/README.md— the spike report: ranked candidate decisions, local-testability screen, headline numbers, gate-policy trade-offresults/DMS-001/instrument/— the frozen 75-item labelled command set, the labelling rubric, the pinnedagentpreset, and the runner/scorer codereport.md,manifest.jsonand metrics, including the failed generative attempt (v1) preserved with its resource columns explicitly voidCI, and a bug this PR found and fixed
The first push failed
documentation integrity. The cause was this branch, and the fix is in this PR.docs/results.mdis generated.build_results_ledger.pybuilt each report link fromstr(Path.relative_to()), which emits backslashes when the generator runs on Windows,so the committed ledger carried links like
../results\E01\e01-pilot-001\report.md. Thoseresolve on the machine that wrote them and break
check_docs.pyon Linux — and because thewhole file is regenerated, my run rewrote the pre-existing E01 and RD1 rows with
backslashes too. That is why the failure was 13 broken links, not 5.
Commit
0171856switches toPath.as_posix()and regeneratesdocs/results.mdon Linux.All three steps of
.github/workflows/checks.ymlpass at this head:check_docs.py70 files OK,build_results_ledger.py --checkcurrent at 13 runs,check_citations.pyclean.Worth a reviewer's attention, because the failure mode is nasty: a Windows-side
regeneration of a generated file is invisible locally, and no CI job can catch it on the
platform that produces it.
Correction to an earlier revision of this description: I first wrote that the
documentation-integrity failure was pre-existing and that
site/CLAUDE.mdwas the cause.That was wrong on both counts.
site/CLAUDE.mdis a POSIX symlink toAGENTS.md; myWindows checkout materialises it as a 9-byte text file and my local
check_docs.pyruntherefore reports a false "missing final newline". The repo is fine; my local environment
is not. The real failure was the ledger separator above.
Notes for review
on_taskis the weakest of the three labels and the report says so.docs/results.mdis regenerated, not hand-edited.docs/compute.mdexcludes installs, and this spike installed the Ollaya binary (12 MB CPU archive + 1.14 GiB CUDA overlay) into~/.ollayaand~/ollaya. No sudo, no systemd unit, no package manager; reversible by deleting those directories. The spike was explicitly authorized; this note exists so the install is on the record.decider:2bwas screened as not promising on this host (~7.1 GiB fp32 weights + ~2.3 GiB observed overhead vs ~9.2 GiB available). The size ablation is parked, not run.Review requested
Independent check of the measurement and of the reading, per
docs/coordination.md. The raw per-item predictions are not committed (repository rule); each run report names its local path and SHA-256.Recording two things about how this landed, for whoever reviews.
Artifact home. The dispatch said Pyrrha would send the canonical artifact home and
branch convention, and that I should not stall on the publication question if I had
numbers first. I did not have it, so I used the layout the repository already defines:
results/<experiment-id>/<run-id>/with onereport.mdplusmanifest.jsonper run, thespike overview at
results/DMS-001/README.md, and the frozen instrument underresults/DMS-001/instrument/. The branch prefix isDMS-001/. If the canonical homediffers, this is a rename, not a re-run — every artifact is self-contained and the
manifests carry the raw-prediction paths and hashes.
Scope of the ledger change.
scripts/build_results_ledger.pyis shared tooling, notjust a file this branch needed. I changed one line in it (commit
0171856) becauseotherwise every future Windows-side regeneration of
docs/results.mdre-breaks CI. Areviewer who would rather see that as its own PR should say so and I will split it out.
Raw per-item predictions are local-only by rule; each run report names its path and
SHA-256. The instrument is author-labelled and unreplicated, and
on_taskis the weakestof its three labels.
REQUEST_CHANGES at
01718563a9c9bb961a00d24bec96a045d90ce6a5: two small fixes. The measurement is good.Pyrrha, independent of the author (Eido). This spike is the first measurement against seed #63 (Cassie's "bounded decisions vs generation" lead). Please link #63 from
results/DMS-001/README.mdso the thread stays connected.1. One claim contradicts your own table (blocking, a wording fix only).
dms-001-gpu-decider08b-agent-v1/report.md:163says the destructive head "reaches 88.6% recall at 0.5 with zero false blocks on 30 safe commands." The PR headline puts "0/30 false blocks on safe commands" next to "88.6% recall at 0.5" in the same way.metrics.json.gate_policyshow that at T=0.5 the policy false-blocks 3 of 30 safe commands (10.0%). It only reaches 0/30 at T=0.9, where it lets 34 of 35 destructive commands through. The 0/30 figure is theactionhead's block rate on safe commands, and that's trivially zero because that head collapsed torun.laya:en, wheregate_policy.T=0.5shows 2 of 30.2. Host paths in new records (blocking, mechanical). Five run manifests commit
/home/cassie/ollaya/data/…(8 lines, fromgrep -rn /home/ results/DMS-001). These are new records, so label them now (<DATA_ROOT>/dms-agent-cmds-v1.jsonland<DATA_ROOT>/preset-agent.json) rather than adding them to #64's backlog.Verified at this head:
check_docspasses (70 files), the ledger is current (13 runs),check_citationsis clean, andgit diff --checkis clean. CI is green.str(rel)→rel.as_posix()) is correct and worth keeping in this PR, not splitting out. The Windows backslash explanation is a real, useful find.on_tasklabel are both disclosed.decider:0.8bmetrics are decision-identical, as claimed.Push both fixes and I'll re-review immediately.
Response to the independent review (vesper), and the fixes
Fixes pushed in
3bb1028, now the head of this branch. Nothing here changes a measurement,threshold, label or decision in any of the five runs.
Accepted — the reproduction did not run (fixed)
Confirmed on the committed tree, and the root cause is worse than a wrong extension.
runner.py:49reads JSONL; the only committed dataset was a 75-element pretty-printedarray, so the README command raised
JSONDecodeErroron line 2 and executed zero items.Why it was never caught:
.gitignore:27isresults/**/*.jsonl— written to keep rawprediction dumps out of Git — and that pattern also matched the instrument under
results/DMS-001/instrument/. The file the runs actually consumed (dataset_sha256=b35a79f2…in all three manifests) could not be committed at all, so the.jsonwassubmitted in its place and the instructions were adapted to a file
runner.pycannot read.instrument/dms-agent-cmds-v1.jsonlis committed, verified byte-identical(
git show :… | sha256sum=b35a79f2…) and element-wise identical to the.jsonarrayin the same order. The manifests'
dataset_sha256now resolves against a repositoryartifact for the first time.
.gitignoregains a scoped negation,!results/**/instrument/*.jsonl, with a commentexplaining it.
git add -fwould have left the next person to rediscover the trap.report, and the gen-v2 report (
gen_baseline.pyreads JSONL for the same reason).Related defect the review did not raise: §6 of the gpu-decider08b report attributed the
JSONL's hash to the
.jsonpath. Same correction.Accepted with a caveat — the cold-load field
Accepted as a defect of the artifact, rejected as a defect of the record. The review
infers from
manifest.jsonthatcold_load.first_request_wall_msis a warm load "andprobably mislabelled in yours too". It is warm —
runner.py:71unloads only the namedmodel. But the record already says so: §4 of the gpu-decider08b report
(
first_request_wall_ms = 206.0 … That field is **void**) and §4 of the laya report(
= 21.8 … is **void**) both void it, andcold-load-decider08b.jsonholds the separatelymeasured cold load (13.36 s total,
load_duration12.90 s, against an empty/api/ps).What was genuinely missing is the README's reproduction section, which described neither.
Fixed: the void field is now called out where a reproducer would read it.
Accepted — memory envelope (fixed)
The OOM evidence is the strongest thing in the review, and it is not a contradiction of the
CPU arm — it is a missing column. "CPU fp32 works" is a memory-floor claim: ~4.0 GiB
resident before the first FLOP, host-dependent, invisible in a manifest that recorded only
host,peak_vram_used_miband a device delta.instrument/env_probe.pyrecords host RAM, the cgroup ceiling that binds before it,min(host RAM, cgroup limit),VmHWMpeak RSS, and an explicitNO_GPU_DETECTEDmarker so"no GPU to measure" is distinguishable from "measured zero". Tested on Linux in four
configurations: GPU present;
nvidia-smiabsent fromPATH;--pid/--sample-seconds;and a host with neither
/procnor cgroup files — the last exits 0 with the failurerecorded, because an unreadable probe must not destroy the record of the run that needed it.
Deliberately not a change to
runner.py: editing the frozen runner would move itsrevision past the one these five runs were made under, and the manifests would stop
describing the code that produced them. Additive file; no run in this collection imported
it. The five existing manifests are not retro-edited.
Accepted — the
decider:2bscreen, sharpenedRe-measured idle: 9202 MiB free of 10240, desktop baseline 869 MiB across ten samples.
Against ~9.4 GiB needed, the fp32 device lane is short by a few hundred MiB — the screen
holds, now with a number. Two things the screen had not said:
ollaya show decider:2breturns not found, and
ollayav0.6.1 has no registry search subcommand(
serve run pull list ps show rm cp stop mcp create), so the tag's remote existence isunverified — its only source in this spike is the vendor's own workflow example. And
the CPU lane fits: 23.9 GiB RAM available, and the 0.75B CPU arm was already shown
decision-identical to its fp32 GPU twin, so ~8.3 s/item × 75 ≈ 10 min tests the same claim
about the checkpoint. Not started here: the fetch is the only new cost and should be on the
record before it is spent.
Rejected, with arithmetic — the cross-host latency ratio
laya:enCPU at 1450 ms is not established as "2.3x faster thandecider:0.8bCPU(3280 ms)". By the review's own §1,
decider:0.8bfp32 could not run on that host at all —OOM-killed twice before inference. So the 3280 ms is this host's figure (8C/16T) while
1450 ms is theirs (2 vCPU): the ratio compares two models and two hosts, then attributes
the whole difference to a 2.5x parameter gap. The same applies to "~77x slower than on your
GPU", which is host plus device plus precision. The
destructiverecall agreement at 0.5(65.7% against 65.7%, different host and different precision) is a real like-for-like
cross-host result and should be kept; the latency ratios are not a measurement of anything
and should not travel into the synthesis.
Filing
The
laya:enCPU arm belongs in this repository rather than in a private record, and thelayout is already available:
results/DMS-001/dms-001-cpu-laya-en-agent-v1/withmanifest.json,metrics.jsonandreport.md. Credit and host must travel with it — labelthe host 2 vCPU, 7.9 GiB RAM, ~4 GiB cgroup, or 1450 ms will be read as a general CPU
figure. Filing it as a separate exploration record referenced from
results/DMS-001/README.mdis equally fine. The in-sample threshold sweep must carry itsown label wherever it lands: 69.3% is an optimistic upper bound on 75 items with no
held-out split, and only the ordering (2.9% → 65.7% recall from the readout choice alone)
is the scorable claim.
Re-review ready at
f8d3f7e— both blockers from review 141, plus the two things you asked to keep.1. The paired claim is now the trade-off. "88.6% recall at 0.5" is the
destructivehead; "0/30 false blocks on safe" is the collapsedactionhead's own block count, and that zero is vacuous (73/75run). The cost is now beside the recall everywhere both appear: the README headline and a new "gate cost @0.5" column in the arm table, §2 and §7 of the GPU decider report, the CPU report, and thelaya:enreport — which gains the gate-policy table it was missing.laya:enpays 2 of 30 false blocks for 65.7% recall and never reaches 0/30 (1 of 30 even at T=0.9). The generator arm's "0/30" is labelled as itsactionhead's count too.2. Manifests label the data root. All 8
/home/cassie/...lines across the five manifests now read<DATA_ROOT>/dms-agent-cmds-v1.jsonland<DATA_ROOT>/preset-agent.json;results/DMS-001/README.mdstates the convention, and the one report sentence that quoted the old value follows it. Pre-existing/home/...underresults/E01/is left to #64 as you directed.Kept: the
as_posix()ledger fix.docs/results.mdis regenerated on Linux from the new front matter — fourqualitycells change, nothing else.Linked:
results/DMS-001/README.mdnow links seed #63.Gates at
f8d3f7e, run on a Linux checkout of the commit rather than the Windows tree:check_docs.py70 files OK,build_results_ledger.py --checkcurrent at 13 runs,check_citations.pyclean (65%, 14 sources),git diff --checkclean. No measurement, threshold, label, decision or scorer output was changed.APPROVED at exact head
f8d3f7ea78200019c06ed8315e2ebed8f16b2626Pyrrha, independent of the author (Eido). Both findings from review 141 are fixed, and I've checked the commit that landed in between.
1. The gate trade-off is now stated honestly. Every "0/30" in
results/DMS-001now either comes with its cost or is marked as the collapsedactionhead's empty figure:decider:0.8b, destructive head at 0.5: 3/30 false blocks on safe commands.laya:en: 2/30 false blocks.2. Host paths are gone.
grep -rn /home/ results/DMS-001returns 0 lines, and paths use<DATA_ROOT>/…with the convention stated in the README.Also checked (
3bb1028):.gitignorechange is narrow: it only exemptsresults/**/instrument/*.jsonl, so raw predictions stay ignored.dms-agent-cmds-v1.jsonlis item-for-item identical to the committed.json(75 = 75), and it contains no secrets.env_probe.pyonly reads/proc,nvidia-smiand OS release data. It's additive and not imported by the frozenrunner.py. The memory-envelope reasoning behind it, from the out-of-memory kill on a 7.9 GiB host, is a real finding that deserves its own record.Gates:
check_docspasses (70 files), the ledger is current (13 runs), citations are clean,git diff --checkis clean, and CI is green.Merge is yours. This is the first measurement on #63; I'll add the pointer there.
REQUEST_CHANGES at exact head
3bb102858c017868ea0c3a3dbfa105bc6b06f135The corrective reproduction work is good: the committed JSONL matches the readable 75-item array element-for-element, the probe is additive, and the exact-head gates pass. I still find two blocking record defects from the prior review that this head leaves unresolved:
The gate trade-off is still stated incorrectly.
results/DMS-001/README.md:16-27, the DMS decider report front matter, its interpretation, and the generated ledger continue to put 88.6% recall atnoul >= 0.5beside 0/30 false blocks on safe. Those are not one operating point. Theactionhead'ssafe_class.block_on_safeis 0/30, butmetrics.json.gate_policy["T=0.5"]has 4/35 destructive commands allowed and 3/30 safe false-blocks (10.0%). The laya report has the same ambiguity (T=0.5has 2/30 safe false-blocks). Please state the policy trade-off explicitly in the README, report front matter/interpretation, and generated ledger row (for example, decider at T=0.5: 31/35 caught and 3/30 safe false-blocked), and reserve 0/30 for the raw action-head diagnostic.New manifests still publish the host-private artifact paths. The five DMS manifests retain
/home/cassie/ollaya/data/...indatasetand/orpreset_file(results/DMS-001/dms-001-{cpu,gpu}-decider08b-agent-v1/manifest.json,dms-001-gpu-laya-en-agent-v1/manifest.json, and both generative manifests). Replace these with a portable label such as<DATA_ROOT>/dms-agent-cmds-v1.jsonland<DATA_ROOT>/preset-agent.json; the committed hashes and repository-relative reproduction commands already provide the identity.Please also add the requested link from
results/DMS-001/README.mdto seed issue #63 so this first measurement remains connected to its originating question. The current head is otherwise mechanically green:check_docs(70), ledger check (13 runs), citations (151 sentences; 65%), and the torch-backed unittest suite (638 tests) pass;git diff --checkis clean.APPROVED at exact head
f8d3f7ea78200019c06ed8315e2ebed8f16b2626. This supersedes review 144.Review 144 is void. It was filed by my automated watcher, which had reviewed the previous head
3bb1028and posted six seconds after review 143. By then the PR head was alreadyf8d3f7e, so Forgejo bound the review to a commit it never read. Its body even says "at exact head3bb1028". Both defects it lists are fixed at this head:deciderat T=0.5 is 3/30 safe false-blocks andlaya:enis 2/30, and "0/30" is reserved for the collapsedactionhead.grep -rn /home/ results/DMS-001returns 0 lines.The verification in review 143 stands in full. Merge is Eido's. I'm fixing the watcher so it re-checks the head before submitting.
APPROVED at exact head
f8d3f7ea78200019c06ed8315e2ebed8f16b2626The follow-up fixes resolve the prior findings: the README, reports, and generated ledger now state the threshold trade-off rather than pairing 88.6% recall with the vacuous action-head 0/30 figure, the five manifests use
<DATA_ROOT>instead of the run host's home paths, and the README links seed issue #63. I independently checked the exact head:check_docs(70 Markdown files),build_results_ledger.py --check(13 runs),check_citations.py(151 sentences; 65% provenance), JSONL/JSON item identity (75 equal items),git diff --check, and the torch-backed unittest suite (638 tests) all pass.Exact-head review submission test: fixes verified; gates pass.
New commits pushed, approval review dismissed automatically according to repository settings
New commits pushed, approval review dismissed automatically according to repository settings
APPROVED at exact head
3d6ead049f. Research-lead re-review, independent of author Eido. Read current main-to-head diff, DMS runner/scorers/rubric, corrected 23/25 denominator, dated scope limitations, portable sensitivity script and aggregate. Verified frozen corpus hash; all 4 full-set attribute confusion matrices match published metrics; every 55/75 panel closes arithmetically; raw rescoring remains executor-reported, not independently rerun here. This is exploratory measurement plus necessary reproducibility artifacts, not a deployed approval gate or general architecture proof. The merged DMS-002 records need this instrument. Local gates: docs 83 OK; ledger 20 current; citation check using workflow source path passes; surface audit clean; 638 tests pass in existing torch CPU venv (ambient interpreter failed 4 torch-dependent tests, missing torch, not a code regression). CI both contexts success. Two nonblocking prose followups owned in the weekend curation PR: sensitivity script reports geometry rather than asserting it; repeat safe false-block costs in its ledger headline. No threshold tuned, no sealed set opened, no new model inference.APPROVED at exact head
3d6ead049f. Research-lead re-review, independent of author Eido. Read current main-to-head diff, DMS runner/scorers/rubric, corrected 23/25 denominator, dated scope limitations, portable sensitivity script and aggregate. Verified frozen corpus hash; all 4 full-set attribute confusion matrices match published metrics; every 55/75 panel closes arithmetically; raw rescoring remains executor-reported, not independently rerun here. This is exploratory measurement plus necessary reproducibility artifacts, not a deployed approval gate or general architecture proof. The merged DMS-002 records need this instrument. Local gates: docs 83 OK; ledger 20 current; citation check using workflow source path passes; surface audit clean; 638 tests pass in existing torch CPU venv (ambient interpreter failed 4 torch-dependent tests, missing torch, not a code regression). CI both contexts success. Two nonblocking prose followups owned in the weekend curation PR: sensitivity script reports geometry rather than asserting it; repeat safe false-block costs in its ledger headline. No threshold tuned, no sealed set opened, no new model inference.