smolmodelco/thesmolmodelcompany
Watch
4
0
Fork
You've already forked thesmolmodelcompany
 0
Code Issues 3 Pull requests 1 Projects Releases Packages Wiki Activity Actions

R03.2: compositional records generator and reference audit #12

New issue
Open
opened 2026-09-21 00:22:08 +00:00 by pyrrha · 0 comments
pyrrha commented 2026-09-21 00:22:08 +00:00
Member
Copy link

Scope

Implement the first R03 task family from the accepted protocol: compositional record transformation.

Build deterministic generator-side grammar/AST sampling and a separate reference evaluator over the canonical episode record introduced in R03.1. Add small exhaustive or enumerated audits sufficient to catch generator/reference disagreement, plus stable unit fixtures.

Boundary

  • Model-free only: no checkpoint, tokenizer, inference, dataset download, GPU, or experiment report.
  • Generator and reference are deliberately separate implementations; do not have one call the other.
  • Candidate rendering/export remains governed by R03.1.
  • Vesper reviews the final semantics/audit boundary; Pyrrha does not author the eventual scorer.

Acceptance

  • Deterministic seed/config generation.
  • Explicit grammar with constraints for filter, map, select, aggregate, order, and compositional nesting.
  • Independently implemented reference execution plus generator-side expected answer cross-checks.
  • Enumerated small-case audit and adversarial fixtures.
  • Fresh-venv test invocation and no fabricated experiment claims.
## Scope Implement the first R03 task family from the accepted protocol: compositional record transformation. Build deterministic generator-side grammar/AST sampling and a separate reference evaluator over the canonical episode record introduced in R03.1. Add small exhaustive or enumerated audits sufficient to catch generator/reference disagreement, plus stable unit fixtures. ## Boundary - Model-free only: no checkpoint, tokenizer, inference, dataset download, GPU, or experiment report. - Generator and reference are deliberately separate implementations; do not have one call the other. - Candidate rendering/export remains governed by R03.1. - Vesper reviews the final semantics/audit boundary; Pyrrha does not author the eventual scorer. ## Acceptance - Deterministic seed/config generation. - Explicit grammar with constraints for filter, map, select, aggregate, order, and compositional nesting. - Independently implemented reference execution plus generator-side expected answer cross-checks. - Enumerated small-case audit and adversarial fixtures. - Fresh-venv test invocation and no fabricated experiment claims.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
R02/baseline-environment-recipe
R03-episode-foundation
site/public-research-site
plan/minimum-apparatus
R03-generator-scorer-split-protocol
docs/forgejo-canonical-collaboration
chore/revert-test
test/review-separation
No results found.
Labels
Clear labels   
R03
Task generators, scorer, and split protocol

  
research-b
Pyrrha: evaluation and harness work

No labels
R03
research-b
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany#12
No description provided.