WIP: LANTERN synthetic diagnostic policy and training apparatus #185

Draft
pyrrha wants to merge 8 commits from research/lantern-model-20261021 into main
Member

What this adds

  • Compact typed synthetic troubleshooting simulator with explicit observation/action costs and harmful blind repair terminal.
  • Observation-history GRU policy and privileged randomized-probe teacher for imitation training (not RL or learned exploration).
  • Prospective generator/split/evaluation protocol, executable split/action-boundary checks, CPU-only bounded runner, tests and actual optimizer/checkpoint reload smoke.

Validation

  • python3 -m py_compile research/lantern_v1/*.py
  • git diff --check
  • Bounded python3 research/lantern_v1/run_bounded.py tests (23 passed)
  • Bounded 12-step GRU smoke with finite gradients, changed weights, optimizer checkpoint reload and exact output equality.

Limits / work in progress

  • Synthetic environment and supplied typed observation encoding; no real-world troubleshooting claim.
  • Teacher uses hidden faults to make training targets; deployed model sees only public event history.
  • Training and held-panel evaluation are later stages; no final panel has been generated.
  • Checkpoint/evidence remain outside Git at /home/cassie/.hermes/smc-lantern-20261021/evidence/lantern_v1/.

This is intentionally WIP for session 1 of the five-stage project.

Session 3 — autonomous evaluation implementation

Implemented a one-shot held-panel evaluator with a frozen checkpoint digest, closed-loop GRU actions from public observation histories, a same-horizon checklist, and a uniform-posterior greedy information-gain comparator. The prospective panel is 32 seen-composition IID + 8 composition-shift incidents across eight seed clusters (120 policy-episode rows); uncertainty resamples paired seed clusters. No final-panel outputs have been generated yet. Comparator policy details and the 2,000-draw seed bootstrap were fixed in the protocol before exposure.

Validation: isolated bounded python3 research/lantern_v1/run_bounded.py tests passed (29 tests); python3 scripts/check_docs.py passed (470 Markdown files); py_compile and git diff --check passed.

The evaluator intentionally runs all three policies on identical incidents, calls torch.load(..., weights_only=True), and writes an exclusive durable latch before generation to prevent a second held-panel launch. Training remains behavior-cloned imitation, not online exploration.

Session 3 — saved-panel challenge audit

Added a reproducible audit that replays every saved trace and checks raw row coverage, costs, actions, and composition/seed replication. All 120 traces replay consistently. The generator challenge found all eight purported seed clusters identical within every one of the 15 composition × policy groups: seed is episode metadata, while faults are fixed by composition. The bootstrap intervals are therefore degenerate and the intended inferential comparison is uninterpretable; keep finite outcomes descriptive only, do not tune or rerun the exposed panel. Full report: research/lantern_v1/STAGE3_REPORT.md; raw evaluation, audit output and exclusive one-shot latch are in the external evidence directory named there.

Latest verification: isolated bounded suite 31 tests passed, cgroup cleanup confirmed; docs check passed for 471 Markdown files; py_compile and git diff --check passed. One short audit unit produced its verdict but the launcher did not observe its cgroup before exit; preserved and disclosed as an infrastructure-accounting failure. Aggregate ledger remains far below authorized ceilings.

Stage 4 — independent saved-panel challenge

  • Independently reimplemented the registered transition semantics without importing the production simulator; replayed all 120 saved traces and recomputed all stratum/policy summaries (zero mismatches).
  • Found all 15 composition×policy groups contain eight identical traces across seed labels. Finite rows are reproducible descriptively; the inferential panel remains uninterpretable. No new evaluation panel or model inference was run.
  • Validation: 3 dedicated challenge tests; bounded full suite; docs gate. Report: research/lantern_v1/STAGE4_CHALLENGE.md.

Stage 5 — final result

  • Completed the five-stage LANTERN run and added research/lantern_v1/FINAL_REPORT.md with the trained checkpoint and evidence hashes, exact finite-panel descriptives, the deterministic seed-cluster inference failure, supplied-vs-learned boundaries, full resource accounting, and reproduction pointers.
  • Final disposition: the 63,656-parameter GRU trained and ran closed-loop, but the planned independent-seed inference is uninterpretable because each nominal seed repeated identical hidden incidents/traces. No win or real-world competence claim; the v1 panel remains consumed.
  • Final report: https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/src/branch/research/lantern-model-20261021/research/lantern_v1/FINAL_REPORT.md
## What this adds - Compact typed synthetic troubleshooting simulator with explicit observation/action costs and harmful blind repair terminal. - Observation-history GRU policy and privileged randomized-probe teacher for *imitation* training (not RL or learned exploration). - Prospective generator/split/evaluation protocol, executable split/action-boundary checks, CPU-only bounded runner, tests and actual optimizer/checkpoint reload smoke. ## Validation - `python3 -m py_compile research/lantern_v1/*.py` - `git diff --check` - Bounded `python3 research/lantern_v1/run_bounded.py tests` (23 passed) - Bounded 12-step GRU smoke with finite gradients, changed weights, optimizer checkpoint reload and exact output equality. ## Limits / work in progress - Synthetic environment and supplied typed observation encoding; no real-world troubleshooting claim. - Teacher uses hidden faults to make training targets; deployed model sees only public event history. - Training and held-panel evaluation are later stages; no final panel has been generated. - Checkpoint/evidence remain outside Git at `/home/cassie/.hermes/smc-lantern-20261021/evidence/lantern_v1/`. This is intentionally WIP for session 1 of the five-stage project. ## Session 3 — autonomous evaluation implementation Implemented a one-shot held-panel evaluator with a frozen checkpoint digest, closed-loop GRU actions from public observation histories, a same-horizon checklist, and a uniform-posterior greedy information-gain comparator. The prospective panel is 32 seen-composition IID + 8 composition-shift incidents across eight seed clusters (120 policy-episode rows); uncertainty resamples paired seed clusters. No final-panel outputs have been generated yet. Comparator policy details and the 2,000-draw seed bootstrap were fixed in the protocol before exposure. Validation: isolated bounded `python3 research/lantern_v1/run_bounded.py tests` passed (29 tests); `python3 scripts/check_docs.py` passed (470 Markdown files); `py_compile` and `git diff --check` passed. The evaluator intentionally runs all three policies on identical incidents, calls `torch.load(..., weights_only=True)`, and writes an exclusive durable latch before generation to prevent a second held-panel launch. Training remains behavior-cloned imitation, not online exploration. ## Session 3 — saved-panel challenge audit Added a reproducible audit that replays every saved trace and checks raw row coverage, costs, actions, and composition/seed replication. All 120 traces replay consistently. The generator challenge found all eight purported seed clusters identical within every one of the 15 composition × policy groups: seed is episode metadata, while faults are fixed by composition. The bootstrap intervals are therefore degenerate and the intended inferential comparison is *uninterpretable*; keep finite outcomes descriptive only, do not tune or rerun the exposed panel. Full report: `research/lantern_v1/STAGE3_REPORT.md`; raw evaluation, audit output and exclusive one-shot latch are in the external evidence directory named there. Latest verification: isolated bounded suite 31 tests passed, cgroup cleanup confirmed; docs check passed for 471 Markdown files; `py_compile` and `git diff --check` passed. One short audit unit produced its verdict but the launcher did not observe its cgroup before exit; preserved and disclosed as an infrastructure-accounting failure. Aggregate ledger remains far below authorized ceilings. ### Stage 4 — independent saved-panel challenge - Independently reimplemented the registered transition semantics without importing the production simulator; replayed all 120 saved traces and recomputed all stratum/policy summaries (zero mismatches). - Found all 15 composition×policy groups contain eight identical traces across seed labels. Finite rows are reproducible descriptively; the inferential panel remains uninterpretable. No new evaluation panel or model inference was run. - Validation: 3 dedicated challenge tests; bounded full suite; docs gate. Report: `research/lantern_v1/STAGE4_CHALLENGE.md`. ### Stage 5 — final result - Completed the five-stage LANTERN run and added `research/lantern_v1/FINAL_REPORT.md` with the trained checkpoint and evidence hashes, exact finite-panel descriptives, the deterministic seed-cluster inference failure, supplied-vs-learned boundaries, full resource accounting, and reproduction pointers. - Final disposition: the 63,656-parameter GRU trained and ran closed-loop, but the planned independent-seed inference is uninterpretable because each nominal seed repeated identical hidden incidents/traces. No win or real-world competence claim; the v1 panel remains consumed. - Final report: https://durandal.exe.xyz/smolmodelco/thesmolmodelcompany/src/branch/research/lantern-model-20261021/research/lantern_v1/FINAL_REPORT.md
research: build LANTERN synthetic diagnostic policy
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 10s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 9s
f25aeff216
docs: clarify LANTERN action and training design
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 10s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 9s
365b2c729e
fix: make LANTERN checkpoints weights-only reloadable
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 10s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 9s
6b2332c851
fix: bind LANTERN split audit to intended strata
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 10s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 10s
4d94a05ae2
feat: add frozen LANTERN closed-loop evaluation
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 11s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 10s
be1ae62d7c
test: audit LANTERN held panel replication assumptions
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 10s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 10s
52838c1869
audit LANTERN saved panel independently
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 10s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 9s
3280627dd5
docs: deliver LANTERN v1 experiment results
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 10s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 10s
496eb8f895
All checks were successful
checks / select relevant integrity checks (pull_request) Successful in 10s
checks / E01 candidate surface sufficiency (pull_request) Has been skipped
checks / DMS-003 protocol freeze integrity (pull_request) Has been skipped
checks / DMS-003 CPU successor integrity (pull_request) Has been skipped
checks / resource accounting and workflow consistency (pull_request) Has been skipped
checks / documentation integrity (pull_request) Successful in 10s
Required
Details
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin research/lantern-model-20261021:research/lantern-model-20261021
git switch research/lantern-model-20261021
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!185
No description provided.