Deferred: RD1 label-free stopping rule, pending acquisition evidence #48

Closed
opened 2026-09-24 19:50:59 +00:00 by pyrrha · 2 comments
Member

Weekend disposition (2026-09-26): Deprioritized, not refuted. p3-probe-001 failed the learnability precondition; D1 fits a tiny training set, while D2 fresh accuracy is at/below its start-free ceilings. Building a stopping policy on that model would optimize how quickly it guesses. Preserve R-agree and the vacuity finding as design evidence. Reopen only after a cheap fixed-depth development experiment establishes transferable task acquisition (the existing frozen precondition, or an independently reviewed prospective replacement). No stage-2 sweep/halting framework this weekend; this does not rule out adaptive depth in general.


Historical issue (preserved)

Proposal for design review, not for implementation yet. SMC-001 RD1 P3B condition 3 (capability loss) is vacuous under the frozen oracle. Addendum 2026-09-24-D (on #42) records this and Vesper confirmed it independently in review 92. This issue proposes the replacement so the design can be argued before any rescaled result exists. The amendment itself will be a dated addendum in its own PR, after #42 merges.

Constraint (from review 92)

Any policy that runs each example to at least its charged bucket is lossless by construction, and that includes the batch-realizable policy. A replacement only carries information if the stopping decision is made without the oracle, or if the estimand changes. It also has to ship a falsifier showing it is non-zero where a non-oracle rule provably stops short.

What cannot be used

The hop count / difficulty is evaluation metadata only and must never reach the policy (addendum A, "does not change"). So a "predicted depth from hop count" rule is out.

Proposal

Condition 3′ (gating): capability loss of a label-free stopping rule. Frozen rule R-agree: walk the grid 1, 2, 4, 8, 16 and stop at the first grid depth whose argmax prediction equals the previous grid depth's; otherwise run to 16. Charged cost is flops(stop). The recurrence is sequential, so the prefix is shared and the charge is honest. The rule uses only the model's own predictions: no label, no hop count, no oracle.

  • loss′ = acc(T=16) − acc(R-agree), same pooled seeds, episode-clustered bootstrap (the C2 unit). The thresholds are unchanged: point ≤ 0.02, upper bound ≤ 0.05.
  • Its saving is reported beside the oracle's. That makes the pair (saving, loss) the realizable trade, and the oracle stays as the ceiling.

Reported, non-gating: cost-matched accuracy. acc(R-agree) against a fixed-depth model at the grid depth nearest R-agree's mean charged depth. This answers "is adaptive better than just picking a shallower T", which condition 3 never asked.

Evidence it is not vacuous (development split only)

Recomputed from run 001's t16-s*-development.jsonl rows. Development only: no final-split row was used to design this rule, per the dev/final separation in AGENTS.md.

seed acc T=16 acc R-agree loss′ stopped before 16 mean stop depth frozen-oracle loss
13 0.1300 0.1289 +0.0011 98.1% 3.00 0.0000
29 0.1078 0.1367 −0.0289 97.4% 3.15 0.0000
41 0.1156 0.1244 −0.0089 98.8% 2.91 0.0000

The frozen-oracle loss is exactly 0 on real rows, which confirms the identity on data, not just on the 32 patterns. loss′ is non-zero on every seed and the rule stops short on ~98% of episodes. Caveat: these models are at chance (the learnability precondition failed), so the numbers show only that the statistic moves. They say nothing about any effect, and the negative losses are noise at chance.

Questions for review

  1. Is agreement-of-consecutive-grid-depths the right label-free rule, or should it be frozen as a family (e.g. require two consecutive agreements) with one member preregistered as primary?
  2. Should condition 3′ gate, with cost-matched accuracy reported? Or the reverse?
  3. Any objection to the falsifier being: a synthetic trajectory set where R-agree provably stops before a later correction flips the answer (loss′ > 0), plus the exhaustive identity test that stays pinned on the old statistic?

No code, no rescaled run, and no GPU work are implied. Stage 2 remains held by the failed stage-1 probe.

**Weekend disposition (2026-09-26):** Deprioritized, not refuted. p3-probe-001 failed the learnability precondition; D1 fits a tiny training set, while D2 fresh accuracy is at/below its start-free ceilings. Building a stopping policy on that model would optimize how quickly it guesses. Preserve R-agree and the vacuity finding as design evidence. Reopen only after a cheap fixed-depth development experiment establishes transferable task acquisition (the existing frozen precondition, or an independently reviewed prospective replacement). No stage-2 sweep/halting framework this weekend; this does not rule out adaptive depth in general. --- ## Historical issue (preserved) **Proposal for design review, not for implementation yet.** SMC-001 RD1 P3B condition 3 (capability loss) is vacuous under the frozen oracle. Addendum 2026-09-24-D (on #42) records this and Vesper confirmed it independently in review 92. This issue proposes the replacement so the design can be argued **before** any rescaled result exists. The amendment itself will be a dated addendum in its own PR, after #42 merges. ## Constraint (from review 92) Any policy that runs each example to at least its charged bucket is lossless by construction, and that includes the batch-realizable policy. A replacement only carries information if the stopping decision is made **without the oracle**, or if the estimand changes. It also has to ship a falsifier showing it is non-zero where a non-oracle rule provably stops short. ## What cannot be used The hop count / `difficulty` is **evaluation metadata only** and must never reach the policy (addendum A, "does not change"). So a "predicted depth from hop count" rule is out. ## Proposal **Condition 3′ (gating): capability loss of a label-free stopping rule.** Frozen rule *R-agree*: walk the grid `1, 2, 4, 8, 16` and stop at the first grid depth whose argmax prediction equals the previous grid depth's; otherwise run to 16. Charged cost is `flops(stop)`. The recurrence is sequential, so the prefix is shared and the charge is honest. The rule uses **only the model's own predictions**: no label, no hop count, no oracle. - loss′ = acc(T=16) − acc(R-agree), same pooled seeds, episode-clustered bootstrap (the C2 unit). The thresholds are unchanged: point ≤ 0.02, upper bound ≤ 0.05. - Its saving is reported beside the oracle's. That makes the pair (saving, loss) the realizable trade, and the oracle stays as the ceiling. **Reported, non-gating: cost-matched accuracy.** acc(R-agree) against a fixed-depth model at the grid depth nearest R-agree's mean charged depth. This answers "is adaptive better than just picking a shallower T", which condition 3 never asked. ## Evidence it is not vacuous (development split only) Recomputed from run 001's `t16-s*-development.jsonl` rows. Development only: no final-split row was used to design this rule, per the dev/final separation in `AGENTS.md`. | seed | acc T=16 | acc R-agree | loss′ | stopped before 16 | mean stop depth | frozen-oracle loss | |---|---|---|---|---|---|---| | 13 | 0.1300 | 0.1289 | **+0.0011** | 98.1% | 3.00 | 0.0000 | | 29 | 0.1078 | 0.1367 | **−0.0289** | 97.4% | 3.15 | 0.0000 | | 41 | 0.1156 | 0.1244 | **−0.0089** | 98.8% | 2.91 | 0.0000 | The frozen-oracle loss is exactly 0 on real rows, which confirms the identity on data, not just on the 32 patterns. loss′ is non-zero on every seed and the rule stops short on ~98% of episodes. **Caveat:** these models are at chance (the learnability precondition failed), so the numbers show only that the statistic *moves*. They say nothing about any effect, and the negative losses are noise at chance. ## Questions for review 1. Is agreement-of-consecutive-grid-depths the right label-free rule, or should it be frozen as a *family* (e.g. require two consecutive agreements) with one member preregistered as primary? 2. Should condition 3′ gate, with cost-matched accuracy reported? Or the reverse? 3. Any objection to the falsifier being: a synthetic trajectory set where R-agree provably stops before a later correction flips the answer (loss′ > 0), plus the exhaustive identity test that stays pinned on the old statistic? No code, no rescaled run, and no GPU work are implied. Stage 2 remains held by the failed stage-1 probe.
Author
Member

Weekend research-lead disposition (2026-09-26)

Deprioritized, not refuted. p3-probe-001 failed the learnability precondition; D1 fits a tiny training set, while D2 fresh accuracy is at/below its start-free ceilings. Building a stopping policy on that model would optimize how quickly it guesses. Preserve R-agree and the vacuity finding as design evidence. Reopen only after a cheap fixed-depth development experiment establishes transferable task acquisition (the existing frozen precondition, or an independently reviewed prospective replacement). No stage-2 sweep/halting framework this weekend; this does not rule out adaptive depth in general.

## Weekend research-lead disposition (2026-09-26) Deprioritized, not refuted. p3-probe-001 failed the learnability precondition; D1 fits a tiny training set, while D2 fresh accuracy is at/below its start-free ceilings. Building a stopping policy on that model would optimize how quickly it guesses. Preserve R-agree and the vacuity finding as design evidence. Reopen only after a cheap fixed-depth development experiment establishes transferable task acquisition (the existing frozen precondition, or an independently reviewed prospective replacement). No stage-2 sweep/halting framework this weekend; this does not rule out adaptive depth in general.
pyrrha changed title from SMC-001 RD1 P3B: replace vacuous capability-loss condition with a label-free stopping rule (design proposal) to Deferred: RD1 label-free stopping rule, pending acquisition evidence 2026-09-26 04:42:24 +00:00
Author
Member

Reopen path, made concrete (no status change)

The disposition above is unchanged. One addition for whoever revisits this, because the disposition cites a precondition without pointing at the evidence that established it or at the cheapest thing that could discharge it.

Evidence behind the precondition — results/exploration/weekend-edge-lanes/audit.md (committed, read-only audit at SMC main eb3bbcf): the preregistered learnability precondition fails at both tested scales, so no depth-demand or P4 claim is licensed, and D1/D2 support capacity-plus-memorisation rather than depth-demand. That is why this is a design proposal resting on chance-level development data rather than a result, and it is why the ticket is deprioritised rather than refuted.

Cheapest thing that could reopen it — one remaining stage-1 learnability probe at the already-named untested learning rate. That is the only defensible optional spend on this line, and explicitly not an authorization for the 15-run grid or any adaptive-stopping/P4 work. If that probe fails too, the honest conclusion is that this instrument does not show transferable task acquisition at this scale, and the halting work should be re-specified against a task where acquisition is demonstrated first rather than probed again.

Noted in passing, not acted on: the same audit rates needle-router's single frozen-development candidate test as more decision-relevant per CPU slot than implementing RD1 adaptive stopping. That lane stays separate and is bounded in the weekend brief's deferred list.

## Reopen path, made concrete (no status change) The disposition above is unchanged. One addition for whoever revisits this, because the disposition cites a precondition without pointing at the evidence that established it or at the cheapest thing that could discharge it. **Evidence behind the precondition** — `results/exploration/weekend-edge-lanes/audit.md` (committed, read-only audit at SMC main `eb3bbcf`): the preregistered learnability precondition fails at **both** tested scales, so no depth-demand or P4 claim is licensed, and D1/D2 support capacity-plus-memorisation rather than depth-demand. That is why this is a design proposal resting on chance-level development data rather than a result, and it is why the ticket is deprioritised rather than refuted. **Cheapest thing that could reopen it** — one remaining stage-1 learnability probe at the already-named untested learning rate. That is the only defensible optional spend on this line, and explicitly **not** an authorization for the 15-run grid or any adaptive-stopping/P4 work. If that probe fails too, the honest conclusion is that this instrument does not show transferable task acquisition at this scale, and the halting work should be re-specified against a task where acquisition is demonstrated first rather than probed again. **Noted in passing, not acted on:** the same audit rates needle-router's single frozen-development candidate test as more decision-relevant per CPU slot than implementing RD1 adaptive stopping. That lane stays separate and is bounded in the weekend brief's deferred list.
Sign in to join this conversation.
No labels
R03
research-b
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany#48
No description provided.