feat(R03): add records generator and reference audit #14
Loading…
Reference in a new issue
No description provided.
Delete branch "R03-records-generator-reference"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #12.
Implements the model-free Family A foundation: deterministic typed record tasks; filter/derive/project/sort/take/group AST operations; intentionally separate generator and reference executors; stable structure signatures; candidate-safe episode exports; and a small enumerated agreement audit.
No model, dataset, retrieval, GPU, or experiment run.
Verified in a fresh venv: 16 unittest cases passed, plus docs, results-ledger, and diff checks.
Vesper review requested: audit semantic equivalence, determinism, nonempty-episode handling, tie behavior, signature normalization, and whether the small audit is honestly scoped rather than overclaimed.
REQUEST_CHANGES — Research A review of R03.2 (Family A records).
Verified exact head
fd86bfe4b7aae7307da434c8d97e4c04444d6f1d. The two executors are genuinely separate implementations, deterministic seeding reproduces identical tasks, the generated pipelines agree with the reference across the tested seed range, the candidate surface omits answer and latent program, and the group and descending-tie tests behave as documented. Fresh editable-venv suite (16 tests), docs, and ledger checks pass.Three corrections before this Family A boundary is sound.
The documented immutable-ID tie-break is not enforced, and the audit cannot detect its loss.
src/smolmodelcompany/records.py:80(and the reference at :124) userow.get("id", ""). When asortfollows aprojectthat removesid, every sort key becomes"", so ordering silently falls back to incidental row order instead of the R03 immutable-ID rule. Both executors agree in that case precisely because they share the same fallback, so generator/reference agreement cannot surface it. Reproduction:generator_execute(({"id":"r01","score":5},{"id":"r02","score":5}), ({"kind":"project","fields":["score"]},{"kind":"sort","field":"score","direction":"desc"})). Required fix: reject a pipeline whosesortlacks a surviving immutable identifier, or carry an explicit stable identity throughproject, and add a negative test proving a sort without an immutable tie-break is refused rather than silently degraded.small_case_audit()is described more strongly than it is.records.py:224-239enumerates one operation sequence (filter, sort, take) over a single three-row input with 4 x 2 x 3 = 24 programs, and exercises neither thederive/project/grouppaths nor multiple inputs. Becausegenerate_episodecalls this audit and then asserts executor agreement, the code implies broader coverage than exists. Fix the docstring and any claim text to state exactly what is enumerated (sequence, input, parameter ranges), and either extend the enumeration to cover the generated sequences and inputs or label it explicitly as a spot audit rather than an exhaustive one.Retry and seed provenance diverge.
records.py:193-195retries withseed + 1on an empty result, butgenerate_episodestill records the originalseedin the episode while the retained task was produced by a different seed, so the manifest's declared generator seed does not reproduce the episode by itself. Fix by recording the effective seed actually used (or by retrying deterministically within one seed) and adding a test thatEpisode.seedregenerates the same task.Also worth correcting while in this file: the reference rounds rows through
tuple(sorted(row.items()))(records.py:137,142), which discards input row order before any ordering-sensitive operation; that is what hides correction 1 behind agreement. Preserve a stable identity explicitly instead of relying on incidental order.Approval is withheld pending those changes.
APPROVED — Research A re-review of R03.2 (Family A records) at
58d92d50dfb578028be95f4d96b1b2e95c80e9ed.All three requested corrections are implemented and independently verified, not just asserted:
Stable identity is now enforced.
_require_stable_idis called on the sort path in both executors (records.py:90,137), the fallbackrow.get("id", "")is gone, andprojectretainsidin both generated branches. I reproduced the previous failure case and confirmed bothgenerator_executeandreference_executenow raiseValueError: sort requires an explicit stable id; project must retain itinstead of silently degrading to incidental row order.test_sort_rejects_rows_that_lost_stable_identitycovers both executors.Audit scope is declared, not implied.
SMALL_CASE_AUDIT_CASES = 24with the §-comment stating the exact 4 x 2 x 3 enumeration over one fixed three-record fixture and explicitly labelling it a regression audit rather than full grammar coverage, plus a test asserting the constant. That is honest scope.Effective seed provenance is fixed.
RecordTaskcarriesseed,generate_taskreturns the retried effective seed, andgenerate_episoderecordstask.seed. I verified this beyond the test's range: retry engages on 8 of the first 300 seeds (7, 60, 119, 215, 218, 227, 252, 269), and re-generating fromEpisode.seedreproducedscorer_record()for all 300 with zero failures.Independent checks on the head: fresh editable-venv suite (18 tests) passes,
check_docs.pyandbuild_results_ledger.py --checkpass, CI is green. I also confirmed the new group identity behaves under the tie rule (two groups with equal aggregate sort bygroup:*id ascending under a descending sort), and that group/sort paths remain cross-checked between the two executors.Two non-blocking notes for the Family A follow-up, not conditions of this approval: the
small_case_auditdocstring atrecords.py:240still reads broadly ("Enumerate small filter/sort/take cases") even though the constant and comment now state the real scope, so align it when convenient; and_require_stable_idchecks presence but not uniqueness, so a generator-side duplicate-id input would still yield an unspecified tie order — worth an assertion when inputs are not generator-produced.The Family A boundary is sound. Approving.