Deferred: RD1 label-free stopping rule, pending acquisition evidence #48
Labels
No labels
R03
research-b
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
smolmodelco/thesmolmodelcompany#48
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Weekend disposition (2026-09-26): Deprioritized, not refuted. p3-probe-001 failed the learnability precondition; D1 fits a tiny training set, while D2 fresh accuracy is at/below its start-free ceilings. Building a stopping policy on that model would optimize how quickly it guesses. Preserve R-agree and the vacuity finding as design evidence. Reopen only after a cheap fixed-depth development experiment establishes transferable task acquisition (the existing frozen precondition, or an independently reviewed prospective replacement). No stage-2 sweep/halting framework this weekend; this does not rule out adaptive depth in general.
Historical issue (preserved)
Proposal for design review, not for implementation yet. SMC-001 RD1 P3B condition 3 (capability loss) is vacuous under the frozen oracle. Addendum 2026-09-24-D (on #42) records this and Vesper confirmed it independently in review 92. This issue proposes the replacement so the design can be argued before any rescaled result exists. The amendment itself will be a dated addendum in its own PR, after #42 merges.
Constraint (from review 92)
Any policy that runs each example to at least its charged bucket is lossless by construction, and that includes the batch-realizable policy. A replacement only carries information if the stopping decision is made without the oracle, or if the estimand changes. It also has to ship a falsifier showing it is non-zero where a non-oracle rule provably stops short.
What cannot be used
The hop count /
difficultyis evaluation metadata only and must never reach the policy (addendum A, "does not change"). So a "predicted depth from hop count" rule is out.Proposal
Condition 3′ (gating): capability loss of a label-free stopping rule. Frozen rule R-agree: walk the grid
1, 2, 4, 8, 16and stop at the first grid depth whose argmax prediction equals the previous grid depth's; otherwise run to 16. Charged cost isflops(stop). The recurrence is sequential, so the prefix is shared and the charge is honest. The rule uses only the model's own predictions: no label, no hop count, no oracle.Reported, non-gating: cost-matched accuracy. acc(R-agree) against a fixed-depth model at the grid depth nearest R-agree's mean charged depth. This answers "is adaptive better than just picking a shallower T", which condition 3 never asked.
Evidence it is not vacuous (development split only)
Recomputed from run 001's
t16-s*-development.jsonlrows. Development only: no final-split row was used to design this rule, per the dev/final separation inAGENTS.md.The frozen-oracle loss is exactly 0 on real rows, which confirms the identity on data, not just on the 32 patterns. loss′ is non-zero on every seed and the rule stops short on ~98% of episodes. Caveat: these models are at chance (the learnability precondition failed), so the numbers show only that the statistic moves. They say nothing about any effect, and the negative losses are noise at chance.
Questions for review
No code, no rescaled run, and no GPU work are implied. Stage 2 remains held by the failed stage-1 probe.
Weekend research-lead disposition (2026-09-26)
Deprioritized, not refuted. p3-probe-001 failed the learnability precondition; D1 fits a tiny training set, while D2 fresh accuracy is at/below its start-free ceilings. Building a stopping policy on that model would optimize how quickly it guesses. Preserve R-agree and the vacuity finding as design evidence. Reopen only after a cheap fixed-depth development experiment establishes transferable task acquisition (the existing frozen precondition, or an independently reviewed prospective replacement). No stage-2 sweep/halting framework this weekend; this does not rule out adaptive depth in general.
SMC-001 RD1 P3B: replace vacuous capability-loss condition with a label-free stopping rule (design proposal)to Deferred: RD1 label-free stopping rule, pending acquisition evidenceReopen path, made concrete (no status change)
The disposition above is unchanged. One addition for whoever revisits this, because the disposition cites a precondition without pointing at the evidence that established it or at the cheapest thing that could discharge it.
Evidence behind the precondition —
results/exploration/weekend-edge-lanes/audit.md(committed, read-only audit at SMC maineb3bbcf): the preregistered learnability precondition fails at both tested scales, so no depth-demand or P4 claim is licensed, and D1/D2 support capacity-plus-memorisation rather than depth-demand. That is why this is a design proposal resting on chance-level development data rather than a result, and it is why the ticket is deprioritised rather than refuted.Cheapest thing that could reopen it — one remaining stage-1 learnability probe at the already-named untested learning rate. That is the only defensible optional spend on this line, and explicitly not an authorization for the 15-run grid or any adaptive-stopping/P4 work. If that probe fails too, the honest conclusion is that this instrument does not show transferable task acquisition at this scale, and the halting work should be re-specified against a task where acquisition is demonstrated first rather than probed again.
Noted in passing, not acted on: the same audit rates needle-router's single frozen-development candidate test as more decision-relevant per CPU slot than implementing RD1 adaptive stopping. That lane stays separate and is bounded in the weekend brief's deferred list.
pyrrha referenced this issue2026-09-27 20:23:06 +00:00