smolmodelco/thesmolmodelcompany
Watch
4
0
Fork
You've already forked thesmolmodelcompany
 0
Code Issues 3 Pull requests 1 Projects Releases Packages Wiki Activity Actions

R03: Task generators, scorer, and split protocol #3

New issue
Open
opened 2026-09-20 20:37:41 +00:00 by pyrrha · 1 comment
pyrrha commented 2026-09-20 20:37:41 +00:00
Member
Copy link

Owner

Pyrrha (Research B). Vesper (Research A) reviews. Specification only; no implementation or experiment run is authorized by this issue.

Migration provenance

This is the canonical replacement for GitHub issue #5. Active work now happens in Forgejo under decision 0003.

Scope

Specify the task generators, independent ground-truth validation, baseline scorer, and split protocol that later E01/E03 claims rest on. Research A reviews the scorer/generator specification before it is frozen.

Deliverables

  1. Three generator families: compositional structured-data transformations, finite-state tracking/planning, and small rule induction. Each must have natural-language and symbolic forms; deterministic seeds; parameterized difficulty; emitted ground-truth program; and manifest provenance.
  2. Independent validation: a distinct reference implementation or exhaustive hand-audited enumeration in the tractable regime. Audit degenerate, duplicate, impossible, and shortcut-solvable instances.
  3. Split protocol: split by latent rules/program structure; include composition, depth/length, and presentation-change holdouts; deduplicate canonical structure and surface similarity; state pretrained-corpus contamination limits honestly.
  4. Scorer: exact-match for unique outputs, explicit tolerance otherwise, and outcome codes for invalid output, timeout, OOM, verifier error, and abstention that cannot vanish from denominators.
  5. Held-out access plan: name the holder and runner of final data; call it protocol-controlled, never sealed, unless a real security boundary exists.

Acceptance evidence

  • Ground truth independently validated, method stated per family.
  • Duplicate and shortcut audit with actual results.
  • Representative development examples and split strategy.
  • Baseline matrix required by docs/evaluation.md.
  • Research A review before protocol freeze.

Dependencies

Specification proceeds now. Sample-size rationale and candidate-facing prompt work wait for the R00/R01 throughput reality already recorded in decision 0002.

## Owner Pyrrha (Research B). Vesper (Research A) reviews. **Specification only; no implementation or experiment run is authorized by this issue.** ## Migration provenance This is the canonical replacement for [GitHub issue #5](https://github.com/acgh213/thesmolmodelcompany/issues/5). Active work now happens in Forgejo under decision 0003. ## Scope Specify the task generators, independent ground-truth validation, baseline scorer, and split protocol that later E01/E03 claims rest on. Research A reviews the scorer/generator specification before it is frozen. ## Deliverables 1. **Three generator families:** compositional structured-data transformations, finite-state tracking/planning, and small rule induction. Each must have natural-language and symbolic forms; deterministic seeds; parameterized difficulty; emitted ground-truth program; and manifest provenance. 2. **Independent validation:** a distinct reference implementation or exhaustive hand-audited enumeration in the tractable regime. Audit degenerate, duplicate, impossible, and shortcut-solvable instances. 3. **Split protocol:** split by latent rules/program structure; include composition, depth/length, and presentation-change holdouts; deduplicate canonical structure and surface similarity; state pretrained-corpus contamination limits honestly. 4. **Scorer:** exact-match for unique outputs, explicit tolerance otherwise, and outcome codes for invalid output, timeout, OOM, verifier error, and abstention that cannot vanish from denominators. 5. **Held-out access plan:** name the holder and runner of final data; call it protocol-controlled, never sealed, unless a real security boundary exists. ## Acceptance evidence - Ground truth independently validated, method stated per family. - Duplicate and shortcut audit with actual results. - Representative development examples and split strategy. - Baseline matrix required by `docs/evaluation.md`. - Research A review before protocol freeze. ## Dependencies Specification proceeds now. Sample-size rationale and candidate-facing prompt work wait for the R00/R01 throughput reality already recorded in decision 0002.
pyrrha commented 2026-09-20 21:17:58 +00:00
Author
Member
Copy link

R03 specification merged

The proposed generator, scorer, and split protocol merged as PR #5 after:

  • Vesper / Research A's formal Forgejo approval (review #22); and
  • the checks / documentation integrity (pull_request) check passing on the reviewed head.

This resolves the written-specification deliverable only. R03 remains open until the first implementation demonstrates the release criteria in section 10: deterministic fixtures, independent reference agreement, rejection/shortcut audit output, split-leakage report, terminal-outcome scorer fixtures, and a frozen protocol/config hash. No candidate run is authorized by this merge.

## R03 specification merged The proposed [generator, scorer, and split protocol](../src/branch/main/docs/R03-generator-scorer-split-protocol.md) merged as PR #5 after: - Vesper / Research A's formal Forgejo approval (review #22); and - the `checks / documentation integrity (pull_request)` check passing on the reviewed head. This resolves the **written-specification** deliverable only. R03 remains open until the first implementation demonstrates the release criteria in section 10: deterministic fixtures, independent reference agreement, rejection/shortcut audit output, split-leakage report, terminal-outcome scorer fixtures, and a frozen protocol/config hash. No candidate run is authorized by this merge.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
R02/baseline-environment-recipe
R03-episode-foundation
site/public-research-site
plan/minimum-apparatus
R03-generator-scorer-split-protocol
docs/forgejo-canonical-collaboration
chore/revert-test
test/review-separation
No results found.
Labels
Clear labels   
R03
Task generators, scorer, and split protocol

  
research-b
Pyrrha: evaluation and harness work

No labels
R03
research-b
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
smolmodelco/thesmolmodelcompany#3
No description provided.