Fixed context adaptation pilot: replay retention benefit with weak policy acquisition #148

Merged
elsie merged 2 commits from research/adaptation-retention-final-20261019 into main 2026-10-19 07:20:12 +00:00
Member

The fixed context adaptation pilot completed once with all ten continued #139 seeds, both learning arms, 20 steps each (400 updates), and every supplied-structure control. Replay greatly reduced old-task degradation, but neither arm acquired the swapped policy well.

Mean exact trajectory accuracy across the fixed seeds, separately for the held-out N8/N16 graphs:

  • Old task A: baseline 100/100%; zero old-loss 34.19/33.89%; replay 90.60/99.54%.
  • New task B: baseline 4.03/3.23%; zero old-loss 5.36/4.13%; replay 4.40/3.41%.
  • Frozen software remapping: 100% exact and strict on both intact tasks. This is a control using supplied relations, not learned adaptation.

Per-seed/per-graph paired deltas, strict scores, endpoints, ties, diagnostic outputs, training audits, and model/optimizer checkpoints are retained under research/adaptation_retention_final_v1. REPORT.md explains what changed and what is not established. Only two held-out graph clusters; naming views/rows are correlated; no row-level significance claim, learned graph recovery, or general continual-learning claim.

Protocol deviation: the prospective registry omitted two calibration-corrupted supplied relation graphs and one channel-swapped development control pair. A separate deterministic artifact postcheck found zero abstract/public-slot overlap with every final original/diagnostic graph. The frozen registry/panel/results were preserved. This is retrospective disjointness evidence, not complete prospective exclusion; no data filtering or training repeat occurred.

PR147 merged as 54115897d1 with its NOT_ADMITTED receipt and original ledger unchanged. Explicit new allocation 002 is recorded separately (300 process-tree CPU /600 supervised worker-wall seconds, 30-second reserve, one CPU and existing memory/output limits). Valid existing calibration was reused; full 2x admission projection 196.863216 CPU/241.919310 wall fit before final materialization. Source/freeze commit 6feb7464ac; manifest c407d92cef40c87642cf5990109793f2064ebfe5c1c96dc4cc8089d304699c57.

Validation: nine tests passed; fresh independent verifier checked 40 raw files/180,480 model rows and 4,512 pure-oracle rows, checkpoint bytes, frozen dependencies, targets, coverage, exclusions, counters and paired contrasts. Offline evidence has predictions/margins/ties, not full logits or an independent model reproduction. Fresh scientific review crosschecked all 20 seed/graph rows and found no blocker. Documentation:303 files passed; results ledger:31 runs current; final artifact audit:169 historical hashes,19 frozen sources,63 published final files. CRLF evidence bytes preserved; whitespace check passed with CR line endings recognized.

Allocation002 total:117.6606927 CPU/192.822332354 worker-wall seconds. Combined histories:511.2112239 CPU/762.147467946 wall. Final-run worker:99.0170927 CPU/108.301416410 wall. All owned workers reaped; no GPU, installs, paid resources, access expansion or repeated final training. Control-plane writing/review/Git/API time excluded consistently with prior accounting. Stop after publishing this reviewed result; leave this PR unmerged pending assessment.

Closeout decision

Exact head 818bf33587 passed independent artifact/scientific review and required checks and was merged under standing SMC authorization. The 20-update recipe is PARKED; all final panels are spent. PR146 supports fresh exact-labeled short-walk and sampled long-walk capability; PR147 preserves the original NOT_ADMITTED result; PR148 shows a finite replay retention benefit but weak new-policy acquisition, so no positive adaptation claim. All three prospective registry omissions remain deviations; the no-overlap postcheck is retrospective.

One proposed next direction, not started: qualify B-only learning on fresh development-only training/held-out graphs before any further retention comparison. Proposed operational gate: at least95% strict exact nonempty B trajectories for every fixed seed on training and held-out development, with pure-oracle/software-remapping controls, a separately assessed fixed schedule/resource ceiling, and full prospective exclusion including corrupted/swapped exposures. If training fit fails, learning/optimization remains the bottleneck; if only transfer fails, address transfer. Only passing both makes a subsequent retention tradeoff informative. No new experiment or resource allocation follows from this note.

Closeout used no training/inference. Allocation002 final118.1726436 CPU/197.596092737 supervised worker-wall seconds; combined histories511.7231748 CPU/766.921228329 wall. All owned workers reaped.

The fixed context adaptation pilot completed once with all ten continued #139 seeds, both learning arms, 20 steps each (400 updates), and every supplied-structure control. Replay greatly reduced old-task degradation, but neither arm acquired the swapped policy well. Mean exact trajectory accuracy across the fixed seeds, separately for the held-out N8/N16 graphs: - Old task A: baseline 100/100%; zero old-loss 34.19/33.89%; replay 90.60/99.54%. - New task B: baseline 4.03/3.23%; zero old-loss 5.36/4.13%; replay 4.40/3.41%. - Frozen software remapping: 100% exact and strict on both intact tasks. This is a control using supplied relations, not learned adaptation. Per-seed/per-graph paired deltas, strict scores, endpoints, ties, diagnostic outputs, training audits, and model/optimizer checkpoints are retained under research/adaptation_retention_final_v1. REPORT.md explains what changed and what is not established. Only two held-out graph clusters; naming views/rows are correlated; no row-level significance claim, learned graph recovery, or general continual-learning claim. Protocol deviation: the prospective registry omitted two calibration-corrupted supplied relation graphs and one channel-swapped development control pair. A separate deterministic artifact postcheck found zero abstract/public-slot overlap with every final original/diagnostic graph. The frozen registry/panel/results were preserved. This is retrospective disjointness evidence, not complete prospective exclusion; no data filtering or training repeat occurred. PR147 merged as 54115897d1ba4503aaa0e325ff8e15c152419230 with its NOT_ADMITTED receipt and original ledger unchanged. Explicit new allocation 002 is recorded separately (300 process-tree CPU /600 supervised worker-wall seconds, 30-second reserve, one CPU and existing memory/output limits). Valid existing calibration was reused; full 2x admission projection 196.863216 CPU/241.919310 wall fit before final materialization. Source/freeze commit 6feb7464aceef3185c0c5467f202dccfdacaeec1; manifest c407d92cef40c87642cf5990109793f2064ebfe5c1c96dc4cc8089d304699c57. Validation: nine tests passed; fresh independent verifier checked 40 raw files/180,480 model rows and 4,512 pure-oracle rows, checkpoint bytes, frozen dependencies, targets, coverage, exclusions, counters and paired contrasts. Offline evidence has predictions/margins/ties, not full logits or an independent model reproduction. Fresh scientific review crosschecked all 20 seed/graph rows and found no blocker. Documentation:303 files passed; results ledger:31 runs current; final artifact audit:169 historical hashes,19 frozen sources,63 published final files. CRLF evidence bytes preserved; whitespace check passed with CR line endings recognized. Allocation002 total:117.6606927 CPU/192.822332354 worker-wall seconds. Combined histories:511.2112239 CPU/762.147467946 wall. Final-run worker:99.0170927 CPU/108.301416410 wall. All owned workers reaped; no GPU, installs, paid resources, access expansion or repeated final training. Control-plane writing/review/Git/API time excluded consistently with prior accounting. Stop after publishing this reviewed result; leave this PR unmerged pending assessment. ## Closeout decision Exact head 818bf335874fa7bdbc76b20e4e4b14483b8f2766 passed independent artifact/scientific review and required checks and was merged under standing SMC authorization. The 20-update recipe is PARKED; all final panels are spent. PR146 supports fresh exact-labeled short-walk and sampled long-walk capability; PR147 preserves the original NOT_ADMITTED result; PR148 shows a finite replay retention benefit but weak new-policy acquisition, so no positive adaptation claim. All three prospective registry omissions remain deviations; the no-overlap postcheck is retrospective. One proposed next direction, not started: qualify B-only learning on fresh development-only training/held-out graphs before any further retention comparison. Proposed operational gate: at least95% strict exact nonempty B trajectories for every fixed seed on training and held-out development, with pure-oracle/software-remapping controls, a separately assessed fixed schedule/resource ceiling, and full prospective exclusion including corrupted/swapped exposures. If training fit fails, learning/optimization remains the bottleneck; if only transfer fails, address transfer. Only passing both makes a subsequent retention tradeoff informative. No new experiment or resource allocation follows from this note. Closeout used no training/inference. Allocation002 final118.1726436 CPU/197.596092737 supervised worker-wall seconds; combined histories511.7231748 CPU/766.921228329 wall. All owned workers reaped.
Report fixed context adaptation pilot with retention tradeoff and exposure deviation
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 8s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 7s
818bf33587
elsie merged commit d0be08249a into main 2026-10-19 07:20:12 +00:00
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!148
No description provided.