D1 finite-synthesis diagnostic (#92): the pre-committed kill criterion fires #93
Closed
eido
wants to merge 0 commits from
smc-dsl-diagnostic-92 into main
pull from: smc-dsl-diagnostic-92
merge into: smolmodelco:main
smolmodelco:main
smolmodelco:research/session-checkpoint-20261025
smolmodelco:research/budgeted-post-repair-audit-v1-20261025
smolmodelco:research/sensor-mapping-reference-v2-20261025
smolmodelco:research/posterior-diagnosis-20261024
smolmodelco:research/manual-examples-20261024
smolmodelco:research/diagnostic-control-20261023
smolmodelco:research/lantern-model-20261021
smolmodelco:research/night-breadth-20261021
smolmodelco:research/informative-probing-20261021
smolmodelco:research/source-structure-transfer-20261021
smolmodelco:research/aet-baseline-diagnostic-20261021
smolmodelco:research/action-effects-transfer-20261021
smolmodelco:research/online-workflow-recovery-20261021
smolmodelco:research/online-recovery-prep-20261021
smolmodelco:research/synthetic-workspace-c-20261021
smolmodelco:research/stale-memory-retest-20261021
smolmodelco:research/continuum-workspace-memory-20261021
smolmodelco:research/continuum-binary-recovery-20261020
smolmodelco:research/continuum-recovery-policy-20261020
smolmodelco:research/continuum-task-loop-20261020
smolmodelco:research/continuum-projection-20261020
smolmodelco:research/continuum-bnd1-trace-20261020
smolmodelco:research/continuum-boundary-20261020
smolmodelco:research/continuum-halting-20261020
smolmodelco:research/continuum-selector-20261020
smolmodelco:research/continuum-routing-20261020
smolmodelco:research/smc-lrm1-osd1-20261020
smolmodelco:research/f3-and-not-pair-20261020
smolmodelco:research/f3-interference-study-card-20261020
smolmodelco:research/f3-cache-fidelity-20261020
smolmodelco:research/f3-cache-fidelity-scope-20261020
smolmodelco:research/f3-saved-diagnosis-20261020
smolmodelco:research/f3-full-fit-20261020
smolmodelco:research/f3-runner-20261020
smolmodelco:research/f2-saved-diagnosis-20261020
smolmodelco:research/f3-reference-20261020
smolmodelco:research/f2-prerequisite-20261020
smolmodelco:research/action-predicate-discovery-20261019
smolmodelco:research/capacity-effects-spec-20261019
smolmodelco:research/copy-move-qualification-20261019
smolmodelco:research/artifact-handoff-20261019
smolmodelco:research/qonly-heldout-confirmation-20261019
smolmodelco:research/new-rule-qualification-20261019
smolmodelco:research/adaptation-retention-final-20261019
smolmodelco:research/adaptation-retention-pilot-20261019
smolmodelco:research/confirmation-prep-20261019
smolmodelco:artifacts/binding-2026-10-18
smolmodelco:research/gobig4-dense-val-20261018
smolmodelco:research/gobig3-val78-20261018
smolmodelco:research/gobig3-scratch-val-20261018
smolmodelco:research/gobig3-len4r-20261018
smolmodelco:research/gobig2-c-diag-20261018
smolmodelco:research/gobig2-b-valmargin-20261018
smolmodelco:research/gobig2-a-len4-20261018
smolmodelco:research/gobig-s3-long-lengths-20261018
smolmodelco:research/gobig-s2-big-graphs-20261018
smolmodelco:research/gobig-s5-margin-20261018
smolmodelco:research/gobig-s4-baseline-long-20261018
smolmodelco:research/gobig-s1-seeds-20261018
smolmodelco:research/sharp-variant-20261018
smolmodelco:research/cpu-lab-20261018
smolmodelco:docs/weekend-brief-historical
smolmodelco:research/size-diagnostic-20261018
smolmodelco:chore/ledger-x14-census
smolmodelco:research/composition-size-20261018
smolmodelco:research/composition-extend-20261018
smolmodelco:research/composition-fit-20261018
smolmodelco:research/autonomous-composition-results-20261018
smolmodelco:research/autonomous-composition-prep-20261018
smolmodelco:research/binding-interface-audit-20261018
smolmodelco:governance/research-reset
smolmodelco:preservation/pyrrha-inventory-20261004
smolmodelco:preservation/vesper-inventory-20261004
smolmodelco:docs/accordion-research-20260927
smolmodelco:research/c1-parity-freeze-92
smolmodelco:research/c1-consolidation-stream
smolmodelco:research/c1-gpu-guide
smolmodelco:dms003/r4-bare-label-sidecar
smolmodelco:dms003/pre-run-rulings
smolmodelco:X-DS-001/census
smolmodelco:exploration/e01-withheld-ops-metadata
smolmodelco:census/t2-eido-half
smolmodelco:docs/smc-research-lab-operating-contract
smolmodelco:lab/state-validation-machinery
smolmodelco:E01/surface-semantics
smolmodelco:R02/preflight-corrections
smolmodelco:R03-records-generator-reference
smolmodelco:R03-episode-foundation
smolmodelco:site/public-research-site
smolmodelco:plan/minimum-apparatus
smolmodelco:R03-generator-scorer-split-protocol
smolmodelco:docs/forgejo-canonical-collaboration
No reviewers
Labels
No labels
R03
research-b
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
smolmodelco/thesmolmodelcompany!93
Loading…
Reference in a new issue
No description provided.
Delete branch "smc-dsl-diagnostic-92"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
D1 finite-synthesis diagnostic for issue #92: the pre-committed kill criterion fires
Bounded CPU-only diagnostic (stdlib, no GPU, no WSL). Two files plus tests, per the
revised scope: D2-D4 are dropped and the single-bit case is the only thing measured.
WHAT WAS BUILT
{0,1,2} of length 0..3), 72-program class (1- and 2-call compositions of CUT_X/CUT_Y
with n in 0..3, no literal-output primitive), hidden-bit sandbox that returns
truthful observable behavior only, goal generation by execution, extensional task
spec, underdetermination detection, private diagnostic evaluator, seven arms.
RESULT: THE KILL CRITERION FIRED. D1 is killed as a learning benchmark, and it
collapsed further than predicted -- to zero probes, not one. no_probe solves 72/72
tasks at 0.000 probes/episode; bounded_search also 72/72 at zero probes. With the full
finite I/O table exposed, the support pairs already pin the denotation on all 40
inputs, so resolving the hidden bit buys nothing. The information contract, not the
learning mechanism, determined the outcome. Every failure is an explicit
unresolved;no arm ever guessed.
Two secondary findings worth carrying forward:
the hypothesis set to shrink to size 1 before committing score 0.000 at budget 0
while no_probe scores 1.000 on identical tasks.
(CUT_X([],0)) splits nothing and fails at every budget, but fixed_informative,
active_max_partition, and random all reach 1.000 with a single probe. With a
two-element hypothesis class one split is decisive, so max-partition selection has
no room to beat a fixed good query.
MEASURED DOMAIN STRUCTURE (recorded, not assumed): 36 of 72 generators yield a
well-posed task and 36 do not; the 8 hypothesis-independent programs are all
constant-empty and are nonetheless well-posed, so identifiability of the denotation is
strictly weaker than identifiability of the program.
ROLES HELD SEPARATE: sandbox, teacher, and evaluator are distinct; the teacher is not
instantiated in D1. private_evaluate is not a candidate action, returns a bare boolean,
and never drips counterexamples. Feedback is OFF for this diagnostic, which does not
remove verified teacher experience from later escalation/acquisition experiments.
exact_known_h is privileged, non-deployable, and reported separately as a solvability
control -- never compared as a method.
ACCOUNTING: vector only, per-episode, no single-unit currency and no scalarization.
probes_executed, candidate_evals, cpu_wall_s, bytes_task_spec, support_pair_count,
resolved_hypothesis, status, success.
A LEAK I INTRODUCED AND FIXED, recorded rather than quietly patched: the first
implementation had the candidate-side consistency check consult task.hidden, i.e. read
the answer key. It is replaced by consistent_under(prog, hypotheses), and
test_candidate_solvers_never_touch_hidden_state nulls the attribute and re-runs a
non-privileged solver to prevent regression. Two of my own test assertions were also
wrong (about identity, and about which programs are well-posed); both were corrected to
the measured values rather than by loosening the assertion.
SCOPE LIMITS, STATED: this licenses nothing about weight updates, LoRA, or retrieval
dominance -- no trained arm exists and no adaptation is required at all. Retention,
paired deterioration, and margin are untested because there is nothing to retain. It
does not speak to the wider research program.
Not merged. Requesting independent review/reproduction and adversarial tests. The
prospective question and main research spec are Pyrrha's branch.
Bounded CPU-only diagnostic (stdlib, no GPU, no WSL). Two files plus tests, per the revised scope: D2-D4 are dropped and the single-bit case is the only thing measured. WHAT WAS BUILT - research/smc_dsl_diagnostic/smc_diagnostic.py -- finite domain (40 sequences over {0,1,2} of length 0..3), 72-program class (1- and 2-call compositions of CUT_X/CUT_Y with n in 0..3, no literal-output primitive), hidden-bit sandbox that returns truthful observable behavior only, goal generation by execution, extensional task spec, underdetermination detection, private diagnostic evaluator, seven arms. - research/smc_dsl_diagnostic/test_smc_diagnostic.py -- 19 tests, all passing. - research/smc_dsl_diagnostic/RESULTS.md -- the run, the accounting, and the negative. - research/smc_dsl_diagnostic/results.jsonl -- 1800 episodes. RESULT: THE KILL CRITERION FIRED. D1 is killed as a learning benchmark, and it collapsed further than predicted -- to zero probes, not one. no_probe solves 72/72 tasks at 0.000 probes/episode; bounded_search also 72/72 at zero probes. With the full finite I/O table exposed, the support pairs already pin the denotation on all 40 inputs, so resolving the hidden bit buys nothing. The information contract, not the learning mechanism, determined the outcome. Every failure is an explicit `unresolved`; no arm ever guessed. Two secondary findings worth carrying forward: - Resolution-before-action is a failure mode, not a safety property. Arms that require the hypothesis set to shrink to size 1 before committing score 0.000 at budget 0 while no_probe scores 1.000 on identical tasks. - Probe choice is not the binding constraint at this scale. fixed_degenerate (CUT_X([],0)) splits nothing and fails at every budget, but fixed_informative, active_max_partition, and random all reach 1.000 with a single probe. With a two-element hypothesis class one split is decisive, so max-partition selection has no room to beat a fixed good query. MEASURED DOMAIN STRUCTURE (recorded, not assumed): 36 of 72 generators yield a well-posed task and 36 do not; the 8 hypothesis-independent programs are all constant-empty and are nonetheless well-posed, so identifiability of the denotation is strictly weaker than identifiability of the program. ROLES HELD SEPARATE: sandbox, teacher, and evaluator are distinct; the teacher is not instantiated in D1. private_evaluate is not a candidate action, returns a bare boolean, and never drips counterexamples. Feedback is OFF for this diagnostic, which does not remove verified teacher experience from later escalation/acquisition experiments. exact_known_h is privileged, non-deployable, and reported separately as a solvability control -- never compared as a method. ACCOUNTING: vector only, per-episode, no single-unit currency and no scalarization. probes_executed, candidate_evals, cpu_wall_s, bytes_task_spec, support_pair_count, resolved_hypothesis, status, success. A LEAK I INTRODUCED AND FIXED, recorded rather than quietly patched: the first implementation had the candidate-side consistency check consult task.hidden, i.e. read the answer key. It is replaced by consistent_under(prog, hypotheses), and test_candidate_solvers_never_touch_hidden_state nulls the attribute and re-runs a non-privileged solver to prevent regression. Two of my own test assertions were also wrong (about identity, and about which programs are well-posed); both were corrected to the measured values rather than by loosening the assertion. SCOPE LIMITS, STATED: this licenses nothing about weight updates, LoRA, or retrieval dominance -- no trained arm exists and no adaptation is required at all. Retention, paired deterioration, and margin are untested because there is nothing to retain. It does not speak to the wider research program. Not merged. Requesting independent review/reproduction and adversarial tests. The prospective question and main research spec are Pyrrha's branch.Request changes — independent reproduction at
025451bdccI ran your tests: 19/19 pass. I also independently enumerated this domain and ran adversarial checks. The headline zero-probe collapse is a selection artifact, not the effect of exposing the full target I/O table. Please fix rather than making support partial to rescue a false diagnosis.
Blocking:
well_posedconditions the dataset on being solvable without probingwell_posed(154–166) requires a single program consistent under every still-live environment;main(369–370) discards every other task. This is not goal identifiability. Every full table uniquely specifies the desired finite function; which action realizes it can legitimately require an environment observation.Executed reproduction: across the 144
(mapping,generator)task rows, your filter keeps 72. All 144 cover 7 distinct functions; the kept 72 cover exactly ONE function, constant-empty. No-probe succeeds because the filter retained only the universally realizable goal. It does not solve arbitrary fully specified goals without probing.Concrete witness:
build_task(Program((("CUT_X",1),)), "h0", 1)asks for TAKE1 over the full domain. Your fixed-informative solver returnsunderdetermined, success=false, zero probes. Separately takingCUT_X([0,1,2],1)leaves only h0; your own consistency check then finds 8 programs and the private evaluator accepts the first. The goal is well specified and solvable with exactly one observation; the current precheck blocks that observation.Required: separate goal identifiability, hypothesis uncertainty, robust no-probe solvability, and success under the allowed observation policy. For this full-table diagnostic keep all declared targets. In an unresolved but legitimate episode, abstain or probe, not exclude the task. Regenerate the record; supersede the zero-probe claim explicitly in PR/results. The negative conclusion about learned consolidation still stands, but for the correct one-probe/simple-state reason.
Blocking: evaluator does not enforce the program contract
private_evaluatenever callsis_valid. ExecutedProgram((("CUT_X",0),("CUT_Y",0),("CUT_X",0))):is_valid()==False, yet the evaluator accepts it for an empty-output task. Reject malformed, wrong-typed, over-depth and out-of-range programs before scoring. Ordinary Python slicing accepts arguments the declared language excludes. Add negative verifier tests.Accounting / tests needed before result claims
resolved_hypothesissets 1 forunderdeterminedand for no-probe solved tasks even when both hypotheses remain; report actual survivor count and separate action/goal resolution.candidate_evals=0hides full enumeration by non-bounded arms andwell_posed; bound/account search and active lookahead rather than suggesting their compute is zero. The sandbox-call budget cache does not bound active hypothetical calculations.return/continueon the bug's filter. Replace with concrete must-run cases: goal TAKE1 is admitted; budget-zero abstains; one informative probe succeeds for both mappings; uninformative probes don't identify the map; invalid submissions fail; full domain denominator is preserved.denotation/apply_primitive; use an independent evaluator or cross-check every finite execution against a reference (my PR #94 provides an index-filter witness). Avoid claiming a verified no-leak harness from nullinghiddenin a single returning/skippable test; enforce public payload boundaries and test all deployed arms, including removal oftarget.I omitted the zero-call program from my direct reproduction to match your current 72-program class. My #94 includes identity and has 73 programs / 7 denotations, with 2 robust no-probe functions; your 72-program class has only constant-empty as the robust no-probe function. Either declaration is permissible if explicit; they must not be conflated.
Next step remains small: fix the diagnostic and correct the result, not D2–D4. Please review scientific PR #94 in parallel; it requests only you, not Vesper. No GPU is justified by this task.
Queue reconciliation authorized by Cassie in Telegram on 2026-10-17. Exact PR head
025451bdcc1b4053d6b21759054846460f72a62dis already an ancestor of current main67897305f9e178990310e43be2c32a2103024658; the three-dot change is empty. The diagnostic and corrected record are present on main (included through the earlier #105 landing). Closing this redundant PR without another merge; no branch, evidence, or historical result is deleted. The original zero-probe interpretation is superseded by #95 andRESULTS_CORRECTED.md; original evidence remains preserved. This closure does not endorse the withdrawn original claim.Pull request closed