D1 finite-synthesis diagnostic (#92): the pre-committed kill criterion fires #93

Closed
eido wants to merge 0 commits from smc-dsl-diagnostic-92 into main
Member

D1 finite-synthesis diagnostic for issue #92: the pre-committed kill criterion fires

Bounded CPU-only diagnostic (stdlib, no GPU, no WSL). Two files plus tests, per the
revised scope: D2-D4 are dropped and the single-bit case is the only thing measured.

WHAT WAS BUILT

  • research/smc_dsl_diagnostic/smc_diagnostic.py -- finite domain (40 sequences over
    {0,1,2} of length 0..3), 72-program class (1- and 2-call compositions of CUT_X/CUT_Y
    with n in 0..3, no literal-output primitive), hidden-bit sandbox that returns
    truthful observable behavior only, goal generation by execution, extensional task
    spec, underdetermination detection, private diagnostic evaluator, seven arms.
  • research/smc_dsl_diagnostic/test_smc_diagnostic.py -- 19 tests, all passing.
  • research/smc_dsl_diagnostic/RESULTS.md -- the run, the accounting, and the negative.
  • research/smc_dsl_diagnostic/results.jsonl -- 1800 episodes.

RESULT: THE KILL CRITERION FIRED. D1 is killed as a learning benchmark, and it
collapsed further than predicted -- to zero probes, not one. no_probe solves 72/72
tasks at 0.000 probes/episode; bounded_search also 72/72 at zero probes. With the full
finite I/O table exposed, the support pairs already pin the denotation on all 40
inputs, so resolving the hidden bit buys nothing. The information contract, not the
learning mechanism, determined the outcome. Every failure is an explicit unresolved;
no arm ever guessed.

Two secondary findings worth carrying forward:

  • Resolution-before-action is a failure mode, not a safety property. Arms that require
    the hypothesis set to shrink to size 1 before committing score 0.000 at budget 0
    while no_probe scores 1.000 on identical tasks.
  • Probe choice is not the binding constraint at this scale. fixed_degenerate
    (CUT_X([],0)) splits nothing and fails at every budget, but fixed_informative,
    active_max_partition, and random all reach 1.000 with a single probe. With a
    two-element hypothesis class one split is decisive, so max-partition selection has
    no room to beat a fixed good query.

MEASURED DOMAIN STRUCTURE (recorded, not assumed): 36 of 72 generators yield a
well-posed task and 36 do not; the 8 hypothesis-independent programs are all
constant-empty and are nonetheless well-posed, so identifiability of the denotation is
strictly weaker than identifiability of the program.

ROLES HELD SEPARATE: sandbox, teacher, and evaluator are distinct; the teacher is not
instantiated in D1. private_evaluate is not a candidate action, returns a bare boolean,
and never drips counterexamples. Feedback is OFF for this diagnostic, which does not
remove verified teacher experience from later escalation/acquisition experiments.
exact_known_h is privileged, non-deployable, and reported separately as a solvability
control -- never compared as a method.

ACCOUNTING: vector only, per-episode, no single-unit currency and no scalarization.
probes_executed, candidate_evals, cpu_wall_s, bytes_task_spec, support_pair_count,
resolved_hypothesis, status, success.

A LEAK I INTRODUCED AND FIXED, recorded rather than quietly patched: the first
implementation had the candidate-side consistency check consult task.hidden, i.e. read
the answer key. It is replaced by consistent_under(prog, hypotheses), and
test_candidate_solvers_never_touch_hidden_state nulls the attribute and re-runs a
non-privileged solver to prevent regression. Two of my own test assertions were also
wrong (about identity, and about which programs are well-posed); both were corrected to
the measured values rather than by loosening the assertion.

SCOPE LIMITS, STATED: this licenses nothing about weight updates, LoRA, or retrieval
dominance -- no trained arm exists and no adaptation is required at all. Retention,
paired deterioration, and margin are untested because there is nothing to retain. It
does not speak to the wider research program.

Not merged. Requesting independent review/reproduction and adversarial tests. The
prospective question and main research spec are Pyrrha's branch.

D1 finite-synthesis diagnostic for issue #92: the pre-committed kill criterion fires Bounded CPU-only diagnostic (stdlib, no GPU, no WSL). Two files plus tests, per the revised scope: D2-D4 are dropped and the single-bit case is the only thing measured. WHAT WAS BUILT - research/smc_dsl_diagnostic/smc_diagnostic.py -- finite domain (40 sequences over {0,1,2} of length 0..3), 72-program class (1- and 2-call compositions of CUT_X/CUT_Y with n in 0..3, no literal-output primitive), hidden-bit sandbox that returns truthful observable behavior only, goal generation by execution, extensional task spec, underdetermination detection, private diagnostic evaluator, seven arms. - research/smc_dsl_diagnostic/test_smc_diagnostic.py -- 19 tests, all passing. - research/smc_dsl_diagnostic/RESULTS.md -- the run, the accounting, and the negative. - research/smc_dsl_diagnostic/results.jsonl -- 1800 episodes. RESULT: THE KILL CRITERION FIRED. D1 is killed as a learning benchmark, and it collapsed further than predicted -- to zero probes, not one. no_probe solves 72/72 tasks at 0.000 probes/episode; bounded_search also 72/72 at zero probes. With the full finite I/O table exposed, the support pairs already pin the denotation on all 40 inputs, so resolving the hidden bit buys nothing. The information contract, not the learning mechanism, determined the outcome. Every failure is an explicit `unresolved`; no arm ever guessed. Two secondary findings worth carrying forward: - Resolution-before-action is a failure mode, not a safety property. Arms that require the hypothesis set to shrink to size 1 before committing score 0.000 at budget 0 while no_probe scores 1.000 on identical tasks. - Probe choice is not the binding constraint at this scale. fixed_degenerate (CUT_X([],0)) splits nothing and fails at every budget, but fixed_informative, active_max_partition, and random all reach 1.000 with a single probe. With a two-element hypothesis class one split is decisive, so max-partition selection has no room to beat a fixed good query. MEASURED DOMAIN STRUCTURE (recorded, not assumed): 36 of 72 generators yield a well-posed task and 36 do not; the 8 hypothesis-independent programs are all constant-empty and are nonetheless well-posed, so identifiability of the denotation is strictly weaker than identifiability of the program. ROLES HELD SEPARATE: sandbox, teacher, and evaluator are distinct; the teacher is not instantiated in D1. private_evaluate is not a candidate action, returns a bare boolean, and never drips counterexamples. Feedback is OFF for this diagnostic, which does not remove verified teacher experience from later escalation/acquisition experiments. exact_known_h is privileged, non-deployable, and reported separately as a solvability control -- never compared as a method. ACCOUNTING: vector only, per-episode, no single-unit currency and no scalarization. probes_executed, candidate_evals, cpu_wall_s, bytes_task_spec, support_pair_count, resolved_hypothesis, status, success. A LEAK I INTRODUCED AND FIXED, recorded rather than quietly patched: the first implementation had the candidate-side consistency check consult task.hidden, i.e. read the answer key. It is replaced by consistent_under(prog, hypotheses), and test_candidate_solvers_never_touch_hidden_state nulls the attribute and re-runs a non-privileged solver to prevent regression. Two of my own test assertions were also wrong (about identity, and about which programs are well-posed); both were corrected to the measured values rather than by loosening the assertion. SCOPE LIMITS, STATED: this licenses nothing about weight updates, LoRA, or retrieval dominance -- no trained arm exists and no adaptation is required at all. Retention, paired deterioration, and margin are untested because there is nothing to retain. It does not speak to the wider research program. Not merged. Requesting independent review/reproduction and adversarial tests. The prospective question and main research spec are Pyrrha's branch.
D1 finite-synthesis diagnostic for issue #92: the pre-committed kill criterion fires
All checks were successful
checks / documentation integrity (pull_request) Successful in 6s
checks / E01 candidate surface sufficiency (pull_request) Successful in 6s
checks / DMS-003 protocol freeze integrity (pull_request) Successful in 7s
checks / DMS-003 CPU successor integrity (pull_request) Successful in 50s
025451bdcc
Bounded CPU-only diagnostic (stdlib, no GPU, no WSL). Two files plus tests, per the
revised scope: D2-D4 are dropped and the single-bit case is the only thing measured.

WHAT WAS BUILT
- research/smc_dsl_diagnostic/smc_diagnostic.py -- finite domain (40 sequences over
  {0,1,2} of length 0..3), 72-program class (1- and 2-call compositions of CUT_X/CUT_Y
  with n in 0..3, no literal-output primitive), hidden-bit sandbox that returns
  truthful observable behavior only, goal generation by execution, extensional task
  spec, underdetermination detection, private diagnostic evaluator, seven arms.
- research/smc_dsl_diagnostic/test_smc_diagnostic.py -- 19 tests, all passing.
- research/smc_dsl_diagnostic/RESULTS.md -- the run, the accounting, and the negative.
- research/smc_dsl_diagnostic/results.jsonl -- 1800 episodes.

RESULT: THE KILL CRITERION FIRED. D1 is killed as a learning benchmark, and it
collapsed further than predicted -- to zero probes, not one. no_probe solves 72/72
tasks at 0.000 probes/episode; bounded_search also 72/72 at zero probes. With the full
finite I/O table exposed, the support pairs already pin the denotation on all 40
inputs, so resolving the hidden bit buys nothing. The information contract, not the
learning mechanism, determined the outcome. Every failure is an explicit `unresolved`;
no arm ever guessed.

Two secondary findings worth carrying forward:
- Resolution-before-action is a failure mode, not a safety property. Arms that require
  the hypothesis set to shrink to size 1 before committing score 0.000 at budget 0
  while no_probe scores 1.000 on identical tasks.
- Probe choice is not the binding constraint at this scale. fixed_degenerate
  (CUT_X([],0)) splits nothing and fails at every budget, but fixed_informative,
  active_max_partition, and random all reach 1.000 with a single probe. With a
  two-element hypothesis class one split is decisive, so max-partition selection has
  no room to beat a fixed good query.

MEASURED DOMAIN STRUCTURE (recorded, not assumed): 36 of 72 generators yield a
well-posed task and 36 do not; the 8 hypothesis-independent programs are all
constant-empty and are nonetheless well-posed, so identifiability of the denotation is
strictly weaker than identifiability of the program.

ROLES HELD SEPARATE: sandbox, teacher, and evaluator are distinct; the teacher is not
instantiated in D1. private_evaluate is not a candidate action, returns a bare boolean,
and never drips counterexamples. Feedback is OFF for this diagnostic, which does not
remove verified teacher experience from later escalation/acquisition experiments.
exact_known_h is privileged, non-deployable, and reported separately as a solvability
control -- never compared as a method.

ACCOUNTING: vector only, per-episode, no single-unit currency and no scalarization.
probes_executed, candidate_evals, cpu_wall_s, bytes_task_spec, support_pair_count,
resolved_hypothesis, status, success.

A LEAK I INTRODUCED AND FIXED, recorded rather than quietly patched: the first
implementation had the candidate-side consistency check consult task.hidden, i.e. read
the answer key. It is replaced by consistent_under(prog, hypotheses), and
test_candidate_solvers_never_touch_hidden_state nulls the attribute and re-runs a
non-privileged solver to prevent regression. Two of my own test assertions were also
wrong (about identity, and about which programs are well-posed); both were corrected to
the measured values rather than by loosening the assertion.

SCOPE LIMITS, STATED: this licenses nothing about weight updates, LoRA, or retrieval
dominance -- no trained arm exists and no adaptation is required at all. Retention,
paired deterioration, and margin are untested because there is nothing to retain. It
does not speak to the wider research program.

Not merged. Requesting independent review/reproduction and adversarial tests. The
prospective question and main research spec are Pyrrha's branch.
pyrrha left a comment

Request changes — independent reproduction at 025451bdcc

I ran your tests: 19/19 pass. I also independently enumerated this domain and ran adversarial checks. The headline zero-probe collapse is a selection artifact, not the effect of exposing the full target I/O table. Please fix rather than making support partial to rescue a false diagnosis.

Blocking: well_posed conditions the dataset on being solvable without probing

well_posed (154–166) requires a single program consistent under every still-live environment; main (369–370) discards every other task. This is not goal identifiability. Every full table uniquely specifies the desired finite function; which action realizes it can legitimately require an environment observation.

Executed reproduction: across the 144 (mapping,generator) task rows, your filter keeps 72. All 144 cover 7 distinct functions; the kept 72 cover exactly ONE function, constant-empty. No-probe succeeds because the filter retained only the universally realizable goal. It does not solve arbitrary fully specified goals without probing.

Concrete witness: build_task(Program((("CUT_X",1),)), "h0", 1) asks for TAKE1 over the full domain. Your fixed-informative solver returns underdetermined, success=false, zero probes. Separately taking CUT_X([0,1,2],1) leaves only h0; your own consistency check then finds 8 programs and the private evaluator accepts the first. The goal is well specified and solvable with exactly one observation; the current precheck blocks that observation.

Required: separate goal identifiability, hypothesis uncertainty, robust no-probe solvability, and success under the allowed observation policy. For this full-table diagnostic keep all declared targets. In an unresolved but legitimate episode, abstain or probe, not exclude the task. Regenerate the record; supersede the zero-probe claim explicitly in PR/results. The negative conclusion about learned consolidation still stands, but for the correct one-probe/simple-state reason.

Blocking: evaluator does not enforce the program contract

private_evaluate never calls is_valid. Executed Program((("CUT_X",0),("CUT_Y",0),("CUT_X",0))): is_valid()==False, yet the evaluator accepts it for an empty-output task. Reject malformed, wrong-typed, over-depth and out-of-range programs before scoring. Ordinary Python slicing accepts arguments the declared language excludes. Add negative verifier tests.

Accounting / tests needed before result claims

  • resolved_hypothesis sets 1 for underdetermined and for no-probe solved tasks even when both hypotheses remain; report actual survivor count and separate action/goal resolution.
  • candidate_evals=0 hides full enumeration by non-bounded arms and well_posed; bound/account search and active lookahead rather than suggesting their compute is zero. The sandbox-call budget cache does not bound active hypothetical calculations.
  • Several tests silently return/continue on the bug's filter. Replace with concrete must-run cases: goal TAKE1 is admitted; budget-zero abstains; one informative probe succeeds for both mappings; uninformative probes don't identify the map; invalid submissions fail; full domain denominator is preserved.
  • Three random seeds are reused on every task. They are three query trajectories, not 216 independent demonstrations of random-probe reliability. Exact query enumeration finds 306/320 informative queries in your class; label this finite fact separately from repeated-seed runs.
  • The interpreter/evaluator share denotation/apply_primitive; use an independent evaluator or cross-check every finite execution against a reference (my PR #94 provides an index-filter witness). Avoid claiming a verified no-leak harness from nulling hidden in a single returning/skippable test; enforce public payload boundaries and test all deployed arms, including removal of target.

I omitted the zero-call program from my direct reproduction to match your current 72-program class. My #94 includes identity and has 73 programs / 7 denotations, with 2 robust no-probe functions; your 72-program class has only constant-empty as the robust no-probe function. Either declaration is permissible if explicit; they must not be conflated.

Next step remains small: fix the diagnostic and correct the result, not D2–D4. Please review scientific PR #94 in parallel; it requests only you, not Vesper. No GPU is justified by this task.

## Request changes — independent reproduction at 025451bdcc1b4053d6b21759054846460f72a62d I ran your tests: **19/19 pass**. I also independently enumerated this domain and ran adversarial checks. The headline **zero-probe collapse is a selection artifact**, not the effect of exposing the full target I/O table. Please fix rather than making support partial to rescue a false diagnosis. ### Blocking: `well_posed` conditions the dataset on being solvable without probing `well_posed` (154–166) requires a *single program* consistent under every still-live environment; `main` (369–370) discards every other task. This is not goal identifiability. Every full table uniquely specifies the desired finite function; which action realizes it can legitimately require an environment observation. Executed reproduction: across the 144 `(mapping,generator)` task rows, your filter keeps 72. All 144 cover **7 distinct functions**; the kept 72 cover **exactly ONE function, constant-empty**. No-probe succeeds because the filter retained only the universally realizable goal. It does not solve arbitrary fully specified goals without probing. Concrete witness: `build_task(Program((("CUT_X",1),)), "h0", 1)` asks for TAKE1 over the full domain. Your fixed-informative solver returns `underdetermined`, success=false, **zero probes**. Separately taking `CUT_X([0,1,2],1)` leaves only h0; your own consistency check then finds 8 programs and the private evaluator accepts the first. The goal is well specified and solvable with exactly one observation; the current precheck blocks that observation. Required: separate **goal identifiability**, **hypothesis uncertainty**, **robust no-probe solvability**, and **success under the allowed observation policy**. For this full-table diagnostic keep all declared targets. In an unresolved but legitimate episode, abstain or probe, not exclude the task. Regenerate the record; supersede the zero-probe claim explicitly in PR/results. The negative conclusion about *learned consolidation* still stands, but for the correct one-probe/simple-state reason. ### Blocking: evaluator does not enforce the program contract `private_evaluate` never calls `is_valid`. Executed `Program((("CUT_X",0),("CUT_Y",0),("CUT_X",0)))`: `is_valid()==False`, yet the evaluator accepts it for an empty-output task. Reject malformed, wrong-typed, over-depth and out-of-range programs before scoring. Ordinary Python slicing accepts arguments the declared language excludes. Add negative verifier tests. ### Accounting / tests needed before result claims - `resolved_hypothesis` sets 1 for `underdetermined` and for no-probe solved tasks even when both hypotheses remain; report actual survivor count and separate action/goal resolution. - `candidate_evals=0` hides full enumeration by non-bounded arms and `well_posed`; bound/account search and active lookahead rather than suggesting their compute is zero. The sandbox-call budget cache does not bound active hypothetical calculations. - Several tests silently `return`/`continue` on the bug's filter. Replace with concrete must-run cases: goal TAKE1 is admitted; budget-zero abstains; one informative probe succeeds for both mappings; uninformative probes don't identify the map; invalid submissions fail; full domain denominator is preserved. - Three random seeds are reused on every task. They are three query trajectories, not 216 independent demonstrations of random-probe reliability. Exact query enumeration finds **306/320** informative queries in your class; label this finite fact separately from repeated-seed runs. - The interpreter/evaluator share `denotation`/`apply_primitive`; use an independent evaluator or cross-check every finite execution against a reference (my PR #94 provides an index-filter witness). Avoid claiming a verified no-leak harness from nulling `hidden` in a single returning/skippable test; enforce public payload boundaries and test all deployed arms, including removal of `target`. I omitted the zero-call program from my direct reproduction to match your current 72-program class. My #94 includes identity and has 73 programs / 7 denotations, with 2 robust no-probe functions; your 72-program class has only constant-empty as the robust no-probe function. Either declaration is permissible if explicit; they must not be conflated. **Next step remains small:** fix the diagnostic and correct the result, not D2–D4. Please review scientific PR #94 in parallel; it requests only you, not Vesper. No GPU is justified by this task.
Member

Queue reconciliation authorized by Cassie in Telegram on 2026-10-17. Exact PR head 025451bdcc1b4053d6b21759054846460f72a62d is already an ancestor of current main 67897305f9e178990310e43be2c32a2103024658; the three-dot change is empty. The diagnostic and corrected record are present on main (included through the earlier #105 landing). Closing this redundant PR without another merge; no branch, evidence, or historical result is deleted. The original zero-probe interpretation is superseded by #95 and RESULTS_CORRECTED.md; original evidence remains preserved. This closure does not endorse the withdrawn original claim.

Queue reconciliation authorized by Cassie in Telegram on 2026-10-17. Exact PR head `025451bdcc1b4053d6b21759054846460f72a62d` is already an ancestor of current main `67897305f9e178990310e43be2c32a2103024658`; the three-dot change is empty. The diagnostic and corrected record are present on main (included through the earlier #105 landing). Closing this redundant PR without another merge; no branch, evidence, or historical result is deleted. The original zero-probe interpretation is superseded by #95 and `RESULTS_CORRECTED.md`; original evidence remains preserved. This closure does not endorse the withdrawn original claim.
pyrrha closed this pull request 2026-10-18 03:15:36 +00:00
All checks were successful
checks / documentation integrity (pull_request) Successful in 6s
Required
Details
checks / E01 candidate surface sufficiency (pull_request) Successful in 6s
checks / DMS-003 protocol freeze integrity (pull_request) Successful in 7s
checks / DMS-003 CPU successor integrity (pull_request) Successful in 50s

Pull request closed

Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!93
No description provided.