research: dense-validation held-out stop (GOBIG4-DENSE-FROZEN-001) — base #143 #145

Open
starsetbyte wants to merge 2 commits from research/gobig4-dense-val-20261018 into research/gobig3-val78-20261018
Owner

Base: research/gobig3-val78-20261018 (#143, 6f8bbfb). Unmerged; for Codex review. Owner-approved single study (Cassie, 2026-10-18), cap 40 min wall / 30 CPU-min.

Study: dense-validation held-out stop (GOBIG4-DENSE-FROZEN-001)

Recipe B (#139) rerun with a much denser held-out stop set: continue #133 gated_sharp_max checkpoints (same init as #139), 10 seeds 619719-619728, on the original 840 train rows, until train strict (margin >= 0.001) AND VALD all-correct with min margin >= 0.001; check every 5 updates; cap 100 (admitted).

VALD (19,456 rows): 4-node graphs, 4 graphs (seed 863101), per graph/start ALL 480 words of lengths 5-8 (exhaustive, not sampled) + a frozen deterministic sample of 32 distinct words at each length 9-12; base + fresh_names views.

Overlap handling: VALD graphs are drawn with rejection of the 48 PL16 abstract graphs (reconstructed by pl16_graphs(), verified in precheck against every PL16 trace), so no (graph, start, word) triple is shared with an eval panel. Names/keys disjoint from train and all eval panels (asserted at freeze). Word strings necessarily recur in PL16. PN8/PN16 are 8/16-node panels, structurally disjoint.

Framework forked to research/gobig_v3 (gv2.py -> gv3.py) so earlier FROZEN hashes stay valid. Only loop change: VALD is checked only when the train check already passes (same stop decision).

Protocol: stdlib tests -> dummy pipeline -> budget check -> FROZEN (sha256 7b8cfa4d41d85fff43a34ad739df2863b73f148a60efd5890946c31dc0529548) -> calibration -> 2x admission -> fit -> eval, each a separate supervised command (1 CPU, nice +10, 2 GiB RSS); all exit 0.

Receipts (wall / CPU / peak RSS): calibration 11.2 s / 11.2 s / 557 MiB; fit 37.4 s / 37.4 s / 380 MiB; eval 146.9 s / 146.8 s / 483 MiB. Total 195.5 s wall / 195.4 s CPU (of 2400 / 1800).

Dense-validation held-out stop vs #139 and #143

Per seed: strict depth PN8 / PN16 / PL16, PL16 exact at length 16, extra updates over #133.
#143 column: the 6 continued seeds use the val78 PL16 eval; other entries (and all PN8/PN16 for #143, which val78 did not re-run) are #139 values.

seed #139 depth #139 L16 #139 upd #143 depth #143 L16 dense depth dense L16 dense upd
619719 4/4/5 0.9714 5 4/4/15 0.9987 4/4/15 0.9987 10
619720 4/4/5 0.9753 10 4/4/5 0.9753 4/4/5 0.9753 10
619721 4/4/6 0.9870 20 4/4/6 0.9870 4/4/6 0.9870 20
619722 4/4/16 1.0000 10 4/4/16 1.0000 4/4/16 1.0000 10
619723 4/4/5 0.9818 5 4/4/5 0.9818 4/4/16 1.0000 10
619724 4/4/8 0.9688 5 4/4/8 0.9688 4/4/16 1.0000 10
619725 4/4/5 0.9831 10 4/4/5 0.9831 4/4/5 0.9831 10
619726 4/4/15 0.9987 15 4/4/15 0.9987 4/4/5 0.9753 10
619727 4/4/16 1.0000 30 4/4/16 1.0000 3/4/3 0.9479 10
619728 4/4/16 1.0000 10 4/4/16 1.0000 4/4/16 1.0000 10
run PN8 depth4 seeds PN16 depth4 seeds PL16 depth16 seeds PL16 mean exact L8 / L12 / L16 PL16 min exact L16
#139 10/10 10/10 3/10 0.9990 / 0.9914 / 0.9866 0.9688
#143 10/10 10/10 3/10 0.9991 / 0.9934 / 0.9893 0.9688
dense (this PR) 9/10 10/10 4/10 0.9966 / 0.9889 / 0.9867 0.9479

Decisions (dense): {"gated_sharp_max": {"PN8": "MIXED_PARTIAL_COMPOSITION", "PN16": "CONSISTENT_AUTONOMOUS_COMPOSITION_THROUGH_LENGTH_4", "PL16": "MIXED_PARTIAL_COMPOSITION"}}
Decisions (#139): {"gated_sharp_max": {"PN8": "CONSISTENT_AUTONOMOUS_COMPOSITION_THROUGH_LENGTH_4", "PN16": "CONSISTENT_AUTONOMOUS_COMPOSITION_THROUGH_LENGTH_4", "PL16": "MIXED_PARTIAL_COMPOSITION"}}

Interpretation

  • Dense word coverage on 4 graphs did not fix the long-length tail: every seed passed VALD (all 480 length 5-8 words, exact, margin >= 1.0), yet 5 seeds still fail PL16 at lengths 5-8. Remaining errors are graph-instance-dependent, not rare-word-dependent; the stop set needs more graphs, not more words.
  • Gains: 619723 and 619724 reach PL16 depth 16 (more updates than #139). Losses: 619726 (15 -> 5) and 619727, which stopped at 10 updates vs 30 in #139 and regressed to PN8 depth 3 / PL16 depth 3. VALD is 4-node only, so it dropped VAL56's 6/8-node size signal; PN8 is no longer CONSISTENT.
  • Net: dense VALD is not better than #139/#143. Next options: (a) many-graph validation (e.g., 48+ fresh 4-node graphs x lengths 5-8 sampled) plus 6/8-node graphs; (b) VALD ∪ VAL56 as the stop set; (c) continue-until-pass with a minimum-update floor.
**Base: `research/gobig3-val78-20261018` (#143, 6f8bbfb).** Unmerged; for Codex review. Owner-approved single study (Cassie, 2026-10-18), cap 40 min wall / 30 CPU-min. ## Study: dense-validation held-out stop (GOBIG4-DENSE-FROZEN-001) Recipe B (#139) rerun with a much denser held-out stop set: continue #133 `gated_sharp_max` checkpoints (same init as #139), 10 seeds 619719-619728, on the original 840 train rows, until train strict (margin >= 0.001) AND **VALD** all-correct with min margin >= 0.001; check every 5 updates; cap 100 (admitted). **VALD** (19,456 rows): 4-node graphs, 4 graphs (seed 863101), per graph/start **ALL** 480 words of lengths 5-8 (exhaustive, not sampled) + a frozen deterministic sample of 32 distinct words at each length 9-12; base + fresh_names views. **Overlap handling:** VALD graphs are drawn with rejection of the 48 PL16 abstract graphs (reconstructed by `pl16_graphs()`, verified in precheck against every PL16 trace), so no (graph, start, word) triple is shared with an eval panel. Names/keys disjoint from train and all eval panels (asserted at freeze). Word *strings* necessarily recur in PL16. PN8/PN16 are 8/16-node panels, structurally disjoint. Framework forked to `research/gobig_v3` (gv2.py -> gv3.py) so earlier FROZEN hashes stay valid. Only loop change: VALD is checked only when the train check already passes (same stop decision). Protocol: stdlib tests -> dummy pipeline -> budget check -> FROZEN (sha256 `7b8cfa4d41d85fff43a34ad739df2863b73f148a60efd5890946c31dc0529548`) -> calibration -> 2x admission -> fit -> eval, each a separate supervised command (1 CPU, nice +10, 2 GiB RSS); all exit 0. Receipts (wall / CPU / peak RSS): calibration 11.2 s / 11.2 s / 557 MiB; fit 37.4 s / 37.4 s / 380 MiB; eval 146.9 s / 146.8 s / 483 MiB. Total 195.5 s wall / 195.4 s CPU (of 2400 / 1800). # Dense-validation held-out stop vs #139 and #143 Per seed: strict depth PN8 / PN16 / PL16, PL16 exact at length 16, extra updates over #133. #143 column: the 6 continued seeds use the val78 PL16 eval; other entries (and all PN8/PN16 for #143, which val78 did not re-run) are #139 values. | seed | #139 depth | #139 L16 | #139 upd | #143 depth | #143 L16 | dense depth | dense L16 | dense upd | |---|---|---|---|---|---|---|---|---| | 619719 | 4/4/5 | 0.9714 | 5 | 4/4/15 | 0.9987 | 4/4/15 | 0.9987 | 10 | | 619720 | 4/4/5 | 0.9753 | 10 | 4/4/5 | 0.9753 | 4/4/5 | 0.9753 | 10 | | 619721 | 4/4/6 | 0.9870 | 20 | 4/4/6 | 0.9870 | 4/4/6 | 0.9870 | 20 | | 619722 | 4/4/16 | 1.0000 | 10 | 4/4/16 | 1.0000 | 4/4/16 | 1.0000 | 10 | | 619723 | 4/4/5 | 0.9818 | 5 | 4/4/5 | 0.9818 | 4/4/16 | 1.0000 | 10 | | 619724 | 4/4/8 | 0.9688 | 5 | 4/4/8 | 0.9688 | 4/4/16 | 1.0000 | 10 | | 619725 | 4/4/5 | 0.9831 | 10 | 4/4/5 | 0.9831 | 4/4/5 | 0.9831 | 10 | | 619726 | 4/4/15 | 0.9987 | 15 | 4/4/15 | 0.9987 | 4/4/5 | 0.9753 | 10 | | 619727 | 4/4/16 | 1.0000 | 30 | 4/4/16 | 1.0000 | 3/4/3 | 0.9479 | 10 | | 619728 | 4/4/16 | 1.0000 | 10 | 4/4/16 | 1.0000 | 4/4/16 | 1.0000 | 10 | | run | PN8 depth4 seeds | PN16 depth4 seeds | PL16 depth16 seeds | PL16 mean exact L8 / L12 / L16 | PL16 min exact L16 | |---|---|---|---|---|---| | #139 | 10/10 | 10/10 | 3/10 | 0.9990 / 0.9914 / 0.9866 | 0.9688 | | #143 | 10/10 | 10/10 | 3/10 | 0.9991 / 0.9934 / 0.9893 | 0.9688 | | dense (this PR) | 9/10 | 10/10 | 4/10 | 0.9966 / 0.9889 / 0.9867 | 0.9479 | Decisions (dense): {"gated_sharp_max": {"PN8": "MIXED_PARTIAL_COMPOSITION", "PN16": "CONSISTENT_AUTONOMOUS_COMPOSITION_THROUGH_LENGTH_4", "PL16": "MIXED_PARTIAL_COMPOSITION"}} Decisions (#139): {"gated_sharp_max": {"PN8": "CONSISTENT_AUTONOMOUS_COMPOSITION_THROUGH_LENGTH_4", "PN16": "CONSISTENT_AUTONOMOUS_COMPOSITION_THROUGH_LENGTH_4", "PL16": "MIXED_PARTIAL_COMPOSITION"}} ## Interpretation - Dense word coverage on 4 graphs did **not** fix the long-length tail: every seed passed VALD (all 480 length 5-8 words, exact, margin >= 1.0), yet 5 seeds still fail PL16 at lengths 5-8. Remaining errors are **graph-instance**-dependent, not rare-word-dependent; the stop set needs more graphs, not more words. - Gains: 619723 and 619724 reach PL16 depth 16 (more updates than #139). Losses: 619726 (15 -> 5) and 619727, which stopped at 10 updates vs 30 in #139 and regressed to PN8 depth 3 / PL16 depth 3. VALD is 4-node only, so it dropped VAL56's 6/8-node size signal; PN8 is no longer CONSISTENT. - Net: dense VALD is not better than #139/#143. Next options: (a) many-graph validation (e.g., 48+ fresh 4-node graphs x lengths 5-8 sampled) plus 6/8-node graphs; (b) VALD ∪ VAL56 as the stop set; (c) continue-until-pass with a minimum-update floor.
gobig_v3/dense: calibration + admission + fit + eval + comparison vs #139/#143
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 7s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 7s
072321307d
Author
Owner

Status: research paused by Cassie (Oct 18, 2026, 11:56 PM ET). No further runs until she resumes.

Takeaway: the remaining errors depend on which graph is tested, not on rare words. Every seed got every length 5–8 check word right on the 4 check graphs, but 5 seeds still fail on the evaluation graphs. The check set was 4-node only, so it also lost the size signal from #139's 4/6/8-node set.

Next-step options (each needs a new freeze and Cassie's approval):

  1. Many-graph check set: about 48 or more new 4-node graphs with sampled words at lengths 5–8, plus 6- and 8-node graphs.
  2. Combined stop rule: this dense check set plus #139's 4/6/8-node check set.
  3. Minimum extra updates: keep the current stop rule, but require a minimum number of extra updates before training can stop.
**Status: research paused by Cassie (Oct 18, 2026, 11:56 PM ET).** No further runs until she resumes. Takeaway: the remaining errors depend on which graph is tested, not on rare words. Every seed got every length 5–8 check word right on the 4 check graphs, but 5 seeds still fail on the evaluation graphs. The check set was 4-node only, so it also lost the size signal from #139's 4/6/8-node set. **Next-step options** (each needs a new freeze and Cassie's approval): 1. **Many-graph check set:** about 48 or more new 4-node graphs with sampled words at lengths 5–8, plus 6- and 8-node graphs. 2. **Combined stop rule:** this dense check set plus #139's 4/6/8-node check set. 3. **Minimum extra updates:** keep the current stop rule, but require a minimum number of extra updates before training can stop.
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 7s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 7s
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin research/gobig4-dense-val-20261018:research/gobig4-dense-val-20261018
git switch research/gobig4-dense-val-20261018

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch research/gobig3-val78-20261018
git merge --no-ff research/gobig4-dense-val-20261018
git switch research/gobig4-dense-val-20261018
git rebase research/gobig3-val78-20261018
git switch research/gobig3-val78-20261018
git merge --ff-only research/gobig4-dense-val-20261018
git switch research/gobig4-dense-val-20261018
git rebase research/gobig3-val78-20261018
git switch research/gobig3-val78-20261018
git merge --no-ff research/gobig4-dense-val-20261018
git switch research/gobig3-val78-20261018
git merge --squash research/gobig4-dense-val-20261018
git switch research/gobig3-val78-20261018
git merge --ff-only research/gobig4-dense-val-20261018
git switch research/gobig3-val78-20261018
git merge research/gobig4-dense-val-20261018
git push origin research/gobig3-val78-20261018
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!145
No description provided.