Qualified short/long fixed-checkpoint confirmation and verification limits #146

Merged
elsie merged 7 commits from research/confirmation-prep-20261019 into main 2026-10-19 06:42:10 +00:00
Member

The separate long-walk evaluator is implemented, qualified, frozen and executed once. Original SIZE001 evidence and its review corrections remain unchanged. This draft now contains both the preserved short result and complete LONG-WALK-001 evidence.

Long-walk outcome

Three fresh exact-labeled graphs each at N8/N16; all starts; fixed length grid [0,4,8,12,16]; 1/16/16/16/16 words per graph/start/length, with long words uniformly sampled without replacement. Same 20 original #133 / selected #139 checkpoints, zero updates and no reselection.

Strict sampled-grid pass Original Continued
N8 0/10 seeds 9/10
N16 1/10 9/10

Continued seed 619723 fails some trajectories at lengths 12 and 16 at both sizes. Continued L16 exactness: N8 7678/7680; N16 15356/15360. All continued length-four views remain exact. No name-view disagreements. Paired exactness: 6,578 improved,0 regressed, 14 both-wrong views across 93,600 correlated paired views. Six graph instances crossed with ten seed pairs, not independent row trials. Strong sampled long-walk performance is not universal success, exhaustive depth 16, unseen-isomorphism generalization, causal stopping-policy benefit, adaptation or external replication.

Qualification and exact independence scope

21 tests pass, no skips. Pre-import bootstrap verifies all project sources; every checkpoint hashes before development/final CPU load. Original/adapted actual forward outputs match for 40 declared development rows across both sizes/all grid lengths (max error 0, atol 1e-6/rtol 1e-5); same architecture/weights/Torch/host, so adapter qualification rather than independent model replication.

A fresh-context author implemented scoring/public-edge traversal from the contract without reading old scorers. Both reducers agree on all 187,200 live forward-logit rows. A separately authored offline verifier checks all 187,200 saved rows, hashes, coverage, graph provenance, targets and counters. Postchecks confirm 20 checkpoint identities, 144 globally fresh names, zero prior exact graph/name overlap, and unchanged original evidence. Independent artifact reviewers found no counter/report inconsistency.

Correction to earlier verifier claim: it independently recomputed metrics from retained predictions, but reused the generator's node oracle and had no logits. It did not independently reproduce model outputs or rederive argmax/margins/ties. Original construction did already cross-check node traces with a separate public-record traversal. The earlier 'batch/logit parity' phrase overstated tensor/metric tests. New live scoring is stronger, but full logits remain unretained; offline batch digests/check counts are metadata and cannot recover logits. The new offline counter audit also shares the secondary public oracle. These limits are explicit in INDEPENDENCE.md; no claim of fully independent model verification.

Freeze, costs and evidence

Source/protocol/exposure/checkpoint/admission/seed freeze commit 14091da precedes panel generation; SHA256 aacd1f2ebbcdbda40dd104a9f95baebcb3f0647864435ff4a0a75b7c1445db36. Exposure includes all declared prior streams, SIZE001 and own fixtures; N4 remains saturated. Representative development calibration measured CPU and wall separately. Six graphs/size failed 2x admission; three passed at 136.868 CPU / 166.385 wall projected seconds plus safeguards. Final run 139.055 CPU / 149.047 wall; no retry, source tuning, optimizer update or GPU initialization.

Accounted cumulative 366.826 CPU / 484.441 worker wall within original 600/900, one CPU, low priority, 2 GiB sampled group RSS stop, 8 GiB address cap, 32 MiB output / 2 MiB logs. This includes an explicit conservative 5 CPU / 5 wall charge for a brief metadata-only reconstruction outside the guardian; no model/framework call occurred there. Control-plane writing/review/native Git/API remains unmeasured. All owned workers reaped. Sampled group peaks, lock/cleanup receipts, every raw prediction, graph/word/name provenance, per-seed/per-graph tables, reviews and retained failures are committed. 293 Markdown files pass, ledger current, whitespace clean. No training, GPU, paid resources, access expansion, unrelated process control or merge.

Preserved SIZE001 short result: continued 10/10 versus original 2/10 N8 and 1/10 N16 through length 4, on its separate 12 graphs/size. Do not combine those panels' denominators.

Draft for review; no merge requested. Existing Elsie HTTPS identity; only per-command signing disable, global configuration unchanged.

Long report · Independence clarification · Execution receipt

Reviewed artifact-only closeout

Seed 619723 has seven logical failures (14 agreeing name views), clustered in three graphs. First errors occur at steps 10–14; all first wrong outputs select the query/start node and two trajectories end correctly. These are conditional output correlations, not evidence of a hidden reset or forgetting mechanism. No forward inference, model/panel rerun or training occurred in closeout.

See research/confirmation_closeout_v1/REPORT.md and retained FAILURES.json for denominators, verification limits and provenance. Exactly one ADAPTATION_RETENTION_CARD.md is proposed for assessment/admission; it has not been run and does not authorize training. Stop after integration/card.

Fresh independent result/code/card/report review: no integration blocker. Closeout tests: 2 passed; documentation: 295 files passed; results ledger: current at 30 runs. Final accounted cumulative resource totals: 373.3005299 CPU / 517.781080344 supervised worker-wall seconds, including conservative 5/5 upper charge, within the original 600/900 envelope. Control-plane Git/API/edit/review costs excluded. All owned workers reaped; no GPU used in closeout.

Merge under standing human authorization after required exact-head checks pass; no protection bypass.

The separate long-walk evaluator is implemented, qualified, frozen and executed once. Original SIZE001 evidence and its review corrections remain unchanged. This draft now contains both the preserved short result and complete LONG-WALK-001 evidence. ## Long-walk outcome Three fresh exact-labeled graphs each at N8/N16; all starts; fixed length grid [0,4,8,12,16]; 1/16/16/16/16 words per graph/start/length, with long words uniformly sampled without replacement. Same 20 original #133 / selected #139 checkpoints, zero updates and no reselection. | Strict sampled-grid pass | Original | Continued | |---|---:|---:| | N8 | 0/10 seeds | 9/10 | | N16 | 1/10 | 9/10 | Continued seed 619723 fails some trajectories at lengths 12 and 16 at both sizes. Continued L16 exactness: N8 7678/7680; N16 15356/15360. All continued length-four views remain exact. No name-view disagreements. Paired exactness: 6,578 improved,0 regressed, 14 both-wrong views across 93,600 correlated paired views. Six graph instances crossed with ten seed pairs, not independent row trials. Strong sampled long-walk performance is not universal success, exhaustive depth 16, unseen-isomorphism generalization, causal stopping-policy benefit, adaptation or external replication. ## Qualification and exact independence scope 21 tests pass, no skips. Pre-import bootstrap verifies all project sources; every checkpoint hashes before development/final CPU load. Original/adapted actual forward outputs match for 40 declared development rows across both sizes/all grid lengths (max error 0, atol 1e-6/rtol 1e-5); same architecture/weights/Torch/host, so adapter qualification rather than independent model replication. A fresh-context author implemented scoring/public-edge traversal from the contract without reading old scorers. Both reducers agree on all 187,200 live forward-logit rows. A separately authored offline verifier checks all 187,200 saved rows, hashes, coverage, graph provenance, targets and counters. Postchecks confirm 20 checkpoint identities, 144 globally fresh names, zero prior exact graph/name overlap, and unchanged original evidence. Independent artifact reviewers found no counter/report inconsistency. Correction to earlier verifier claim: it independently recomputed metrics from retained predictions, but reused the generator's node oracle and had no logits. It did not independently reproduce model outputs or rederive argmax/margins/ties. Original construction did already cross-check node traces with a separate public-record traversal. The earlier 'batch/logit parity' phrase overstated tensor/metric tests. New live scoring is stronger, but full logits remain unretained; offline batch digests/check counts are metadata and cannot recover logits. The new offline counter audit also shares the secondary public oracle. These limits are explicit in INDEPENDENCE.md; no claim of fully independent model verification. ## Freeze, costs and evidence Source/protocol/exposure/checkpoint/admission/seed freeze commit 14091da precedes panel generation; SHA256 aacd1f2ebbcdbda40dd104a9f95baebcb3f0647864435ff4a0a75b7c1445db36. Exposure includes all declared prior streams, SIZE001 and own fixtures; N4 remains saturated. Representative development calibration measured CPU and wall separately. Six graphs/size failed 2x admission; three passed at 136.868 CPU / 166.385 wall projected seconds plus safeguards. Final run 139.055 CPU / 149.047 wall; no retry, source tuning, optimizer update or GPU initialization. Accounted cumulative 366.826 CPU / 484.441 worker wall within original 600/900, one CPU, low priority, 2 GiB sampled group RSS stop, 8 GiB address cap, 32 MiB output / 2 MiB logs. This includes an explicit conservative 5 CPU / 5 wall charge for a brief metadata-only reconstruction outside the guardian; no model/framework call occurred there. Control-plane writing/review/native Git/API remains unmeasured. All owned workers reaped. Sampled group peaks, lock/cleanup receipts, every raw prediction, graph/word/name provenance, per-seed/per-graph tables, reviews and retained failures are committed. 293 Markdown files pass, ledger current, whitespace clean. No training, GPU, paid resources, access expansion, unrelated process control or merge. Preserved SIZE001 short result: continued 10/10 versus original 2/10 N8 and 1/10 N16 through length 4, on its separate 12 graphs/size. Do not combine those panels' denominators. Draft for review; no merge requested. Existing Elsie HTTPS identity; only per-command signing disable, global configuration unchanged. [Long report](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/src/commit/5bf507be19d33ad63c084107292db0bd5931fe63/research/confirmation_followup_v1/REPORT.md) · [Independence clarification](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/src/commit/5bf507be19d33ad63c084107292db0bd5931fe63/research/confirmation_followup_v1/INDEPENDENCE.md) · [Execution receipt](https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/src/commit/5bf507be19d33ad63c084107292db0bd5931fe63/research/confirmation_followup_v1/evidence/long-walk-001/EXECUTION_RECEIPT.json) ## Reviewed artifact-only closeout Seed 619723 has seven logical failures (14 agreeing name views), clustered in three graphs. First errors occur at steps 10–14; all first wrong outputs select the query/start node and two trajectories end correctly. These are conditional output correlations, not evidence of a hidden reset or forgetting mechanism. No forward inference, model/panel rerun or training occurred in closeout. See research/confirmation_closeout_v1/REPORT.md and retained FAILURES.json for denominators, verification limits and provenance. Exactly one ADAPTATION_RETENTION_CARD.md is proposed for assessment/admission; it has not been run and does not authorize training. Stop after integration/card. Fresh independent result/code/card/report review: no integration blocker. Closeout tests: 2 passed; documentation: 295 files passed; results ledger: current at 30 runs. Final accounted cumulative resource totals: 373.3005299 CPU / 517.781080344 supervised worker-wall seconds, including conservative 5/5 upper charge, within the original 600/900 envelope. Control-plane Git/API/edit/review costs excluded. All owned workers reaped; no GPU used in closeout. Merge under standing human authorization after required exact-head checks pass; no protection bypass.
research: prepare independent binding confirmation without model runs
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 7s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 7s
af0fc9b29e
research: report independently verified fresh size confirmation
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 8s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 8s
9d1711d525
elsie changed title from WIP: Prepare independent binding confirmation (no model runs) to WIP: Fresh N8/N16 fixed-checkpoint confirmation (zero updates) 2026-10-19 05:48:12 +00:00
research: audit immutable confirmation and prepare fresh long-walk protocol
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 8s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 7s
93df1d343b
elsie changed title from WIP: Fresh N8/N16 fixed-checkpoint confirmation (zero updates) to WIP: Fresh size confirmation review and long-walk preparation 2026-10-19 05:59:17 +00:00
research: report qualified long-walk failures and exact verification limits
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 8s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 7s
5bf507be19
elsie changed title from WIP: Fresh size confirmation review and long-walk preparation to WIP: Qualified short/long fixed-checkpoint confirmation and verification limits 2026-10-19 06:25:16 +00:00
Document artifact-only failure closeout and proposed retention study
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 8s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 7s
f27943a361
elsie changed title from WIP: Qualified short/long fixed-checkpoint confirmation and verification limits to Qualified short/long fixed-checkpoint confirmation and verification limits 2026-10-19 06:41:43 +00:00
elsie merged commit 34f8ceaeaa into main 2026-10-19 06:42:10 +00:00
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!146
No description provided.