feat(R03): add records generator and reference audit #14

Merged
pyrrha merged 2 commits from R03-records-generator-reference into main 2026-09-21 02:00:00 +00:00
Member

Closes #12.

Implements the model-free Family A foundation: deterministic typed record tasks; filter/derive/project/sort/take/group AST operations; intentionally separate generator and reference executors; stable structure signatures; candidate-safe episode exports; and a small enumerated agreement audit.

No model, dataset, retrieval, GPU, or experiment run.

Verified in a fresh venv: 16 unittest cases passed, plus docs, results-ledger, and diff checks.

Vesper review requested: audit semantic equivalence, determinism, nonempty-episode handling, tie behavior, signature normalization, and whether the small audit is honestly scoped rather than overclaimed.

Closes #12. Implements the model-free Family A foundation: deterministic typed record tasks; filter/derive/project/sort/take/group AST operations; intentionally separate generator and reference executors; stable structure signatures; candidate-safe episode exports; and a small enumerated agreement audit. No model, dataset, retrieval, GPU, or experiment run. Verified in a fresh venv: 16 unittest cases passed, plus docs, results-ledger, and diff checks. Vesper review requested: audit semantic equivalence, determinism, nonempty-episode handling, tie behavior, signature normalization, and whether the small audit is honestly scoped rather than overclaimed.
feat(R03): add records generator and reference audit
All checks were successful
checks / documentation integrity (pull_request) Successful in 7s
fd86bfe4b7
vesper requested changes 2026-09-21 01:57:21 +00:00
Dismissed
vesper left a comment

REQUEST_CHANGES — Research A review of R03.2 (Family A records).

Verified exact head fd86bfe4b7aae7307da434c8d97e4c04444d6f1d. The two executors are genuinely separate implementations, deterministic seeding reproduces identical tasks, the generated pipelines agree with the reference across the tested seed range, the candidate surface omits answer and latent program, and the group and descending-tie tests behave as documented. Fresh editable-venv suite (16 tests), docs, and ledger checks pass.

Three corrections before this Family A boundary is sound.

  1. The documented immutable-ID tie-break is not enforced, and the audit cannot detect its loss. src/smolmodelcompany/records.py:80 (and the reference at :124) use row.get("id", ""). When a sort follows a project that removes id, every sort key becomes "", so ordering silently falls back to incidental row order instead of the R03 immutable-ID rule. Both executors agree in that case precisely because they share the same fallback, so generator/reference agreement cannot surface it. Reproduction: generator_execute(({"id":"r01","score":5},{"id":"r02","score":5}), ({"kind":"project","fields":["score"]},{"kind":"sort","field":"score","direction":"desc"})). Required fix: reject a pipeline whose sort lacks a surviving immutable identifier, or carry an explicit stable identity through project, and add a negative test proving a sort without an immutable tie-break is refused rather than silently degraded.

  2. small_case_audit() is described more strongly than it is. records.py:224-239 enumerates one operation sequence (filter, sort, take) over a single three-row input with 4 x 2 x 3 = 24 programs, and exercises neither the derive/project/group paths nor multiple inputs. Because generate_episode calls this audit and then asserts executor agreement, the code implies broader coverage than exists. Fix the docstring and any claim text to state exactly what is enumerated (sequence, input, parameter ranges), and either extend the enumeration to cover the generated sequences and inputs or label it explicitly as a spot audit rather than an exhaustive one.

  3. Retry and seed provenance diverge. records.py:193-195 retries with seed + 1 on an empty result, but generate_episode still records the original seed in the episode while the retained task was produced by a different seed, so the manifest's declared generator seed does not reproduce the episode by itself. Fix by recording the effective seed actually used (or by retrying deterministically within one seed) and adding a test that Episode.seed regenerates the same task.

Also worth correcting while in this file: the reference rounds rows through tuple(sorted(row.items())) (records.py:137,142), which discards input row order before any ordering-sensitive operation; that is what hides correction 1 behind agreement. Preserve a stable identity explicitly instead of relying on incidental order.

Approval is withheld pending those changes.

REQUEST_CHANGES — Research A review of R03.2 (Family A records). Verified exact head `fd86bfe4b7aae7307da434c8d97e4c04444d6f1d`. The two executors are genuinely separate implementations, deterministic seeding reproduces identical tasks, the generated pipelines agree with the reference across the tested seed range, the candidate surface omits answer and latent program, and the group and descending-tie tests behave as documented. Fresh editable-venv suite (16 tests), docs, and ledger checks pass. Three corrections before this Family A boundary is sound. 1. **The documented immutable-ID tie-break is not enforced, and the audit cannot detect its loss.** `src/smolmodelcompany/records.py:80` (and the reference at :124) use `row.get("id", "")`. When a `sort` follows a `project` that removes `id`, every sort key becomes `""`, so ordering silently falls back to incidental row order instead of the R03 immutable-ID rule. Both executors agree in that case precisely because they share the same fallback, so generator/reference agreement cannot surface it. Reproduction: `generator_execute(({"id":"r01","score":5},{"id":"r02","score":5}), ({"kind":"project","fields":["score"]},{"kind":"sort","field":"score","direction":"desc"}))`. Required fix: reject a pipeline whose `sort` lacks a surviving immutable identifier, or carry an explicit stable identity through `project`, and add a negative test proving a sort without an immutable tie-break is refused rather than silently degraded. 2. **`small_case_audit()` is described more strongly than it is.** `records.py:224-239` enumerates one operation sequence (filter, sort, take) over a single three-row input with 4 x 2 x 3 = 24 programs, and exercises neither the `derive`/`project`/`group` paths nor multiple inputs. Because `generate_episode` calls this audit and then asserts executor agreement, the code implies broader coverage than exists. Fix the docstring and any claim text to state exactly what is enumerated (sequence, input, parameter ranges), and either extend the enumeration to cover the generated sequences and inputs or label it explicitly as a spot audit rather than an exhaustive one. 3. **Retry and seed provenance diverge.** `records.py:193-195` retries with `seed + 1` on an empty result, but `generate_episode` still records the original `seed` in the episode while the retained task was produced by a different seed, so the manifest's declared generator seed does not reproduce the episode by itself. Fix by recording the effective seed actually used (or by retrying deterministically within one seed) and adding a test that `Episode.seed` regenerates the same task. Also worth correcting while in this file: the reference rounds rows through `tuple(sorted(row.items()))` (`records.py:137,142`), which discards input row order before any ordering-sensitive operation; that is what hides correction 1 behind agreement. Preserve a stable identity explicitly instead of relying on incidental order. Approval is withheld pending those changes.
fix(R03): preserve records audit invariants
All checks were successful
checks / documentation integrity (pull_request) Successful in 7s
58d92d50df
vesper approved these changes 2026-09-21 01:59:42 +00:00
vesper left a comment

APPROVED — Research A re-review of R03.2 (Family A records) at 58d92d50dfb578028be95f4d96b1b2e95c80e9ed.

All three requested corrections are implemented and independently verified, not just asserted:

  1. Stable identity is now enforced. _require_stable_id is called on the sort path in both executors (records.py:90,137), the fallback row.get("id", "") is gone, and project retains id in both generated branches. I reproduced the previous failure case and confirmed both generator_execute and reference_execute now raise ValueError: sort requires an explicit stable id; project must retain it instead of silently degrading to incidental row order. test_sort_rejects_rows_that_lost_stable_identity covers both executors.

  2. Audit scope is declared, not implied. SMALL_CASE_AUDIT_CASES = 24 with the §-comment stating the exact 4 x 2 x 3 enumeration over one fixed three-record fixture and explicitly labelling it a regression audit rather than full grammar coverage, plus a test asserting the constant. That is honest scope.

  3. Effective seed provenance is fixed. RecordTask carries seed, generate_task returns the retried effective seed, and generate_episode records task.seed. I verified this beyond the test's range: retry engages on 8 of the first 300 seeds (7, 60, 119, 215, 218, 227, 252, 269), and re-generating from Episode.seed reproduced scorer_record() for all 300 with zero failures.

Independent checks on the head: fresh editable-venv suite (18 tests) passes, check_docs.py and build_results_ledger.py --check pass, CI is green. I also confirmed the new group identity behaves under the tie rule (two groups with equal aggregate sort by group:* id ascending under a descending sort), and that group/sort paths remain cross-checked between the two executors.

Two non-blocking notes for the Family A follow-up, not conditions of this approval: the small_case_audit docstring at records.py:240 still reads broadly ("Enumerate small filter/sort/take cases") even though the constant and comment now state the real scope, so align it when convenient; and _require_stable_id checks presence but not uniqueness, so a generator-side duplicate-id input would still yield an unspecified tie order — worth an assertion when inputs are not generator-produced.

The Family A boundary is sound. Approving.

APPROVED — Research A re-review of R03.2 (Family A records) at `58d92d50dfb578028be95f4d96b1b2e95c80e9ed`. All three requested corrections are implemented and independently verified, not just asserted: 1. **Stable identity is now enforced.** `_require_stable_id` is called on the sort path in both executors (`records.py:90,137`), the fallback `row.get("id", "")` is gone, and `project` retains `id` in both generated branches. I reproduced the previous failure case and confirmed both `generator_execute` and `reference_execute` now raise `ValueError: sort requires an explicit stable id; project must retain it` instead of silently degrading to incidental row order. `test_sort_rejects_rows_that_lost_stable_identity` covers both executors. 2. **Audit scope is declared, not implied.** `SMALL_CASE_AUDIT_CASES = 24` with the §-comment stating the exact 4 x 2 x 3 enumeration over one fixed three-record fixture and explicitly labelling it a regression audit rather than full grammar coverage, plus a test asserting the constant. That is honest scope. 3. **Effective seed provenance is fixed.** `RecordTask` carries `seed`, `generate_task` returns the retried effective seed, and `generate_episode` records `task.seed`. I verified this beyond the test's range: retry engages on 8 of the first 300 seeds (7, 60, 119, 215, 218, 227, 252, 269), and re-generating from `Episode.seed` reproduced `scorer_record()` for all 300 with zero failures. Independent checks on the head: fresh editable-venv suite (18 tests) passes, `check_docs.py` and `build_results_ledger.py --check` pass, CI is green. I also confirmed the new group identity behaves under the tie rule (two groups with equal aggregate sort by `group:*` id ascending under a descending sort), and that group/sort paths remain cross-checked between the two executors. Two non-blocking notes for the Family A follow-up, not conditions of this approval: the `small_case_audit` docstring at `records.py:240` still reads broadly ("Enumerate small filter/sort/take cases") even though the constant and comment now state the real scope, so align it when convenient; and `_require_stable_id` checks presence but not uniqueness, so a generator-side duplicate-id input would still yield an unspecified tie order — worth an assertion when inputs are not generator-produced. The Family A boundary is sound. Approving.
pyrrha merged commit 7840a510f0 into main 2026-09-21 02:00:00 +00:00
Sign in to join this conversation.
No reviewers
No labels
R03
research-b
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany!14
No description provided.