Quality

Review checkpoints and curation

Measure task quality, create a golden set, and improve the next version.

A review checkpoint samples each selected task's decisions independently. It can request automated review, human curation cards, or both.

Workbench and the checkpoint editor default to GPT-5.6 Sol with high reasoning effort. You may choose another reviewer independently of the task's model. Python checkpoints declare the reviewer explicitly; omit provider and model for human-only review.

For automated review of a deterministic task, supply its description: what the task should do. The reviewer also receives the function source and bound values. A task without a description is skipped; if none of the checkpoint's tasks is eligible, that checkpoint fails while other checkpoints continue. Human curation does not require a description.

Review(
    name="event-thread-quality",
    tasks=("group-events",),
    sample_size=100,
    curation_sample_size=30,
    provider="openai",
    model="gpt-5.6-sol",
    reasoning_effort="high",
    trust_reviewer=False,
)

Parameter	Default	Meaning
`name`	required	Stable checkpoint name.
`tasks`	required	Named tasks reviewed independently by this checkpoint.
`sample_size`	50	Unique examples sent for automated review.
`curation_sample_size`	0	Cards prepared for a person. A positive number enables human curation.
`provider / model`	none	Automated review model. Omit for curation-only review.
`max_output_tokens`	8192	Per-request response ceiling for automated review.
`include_reason`	true	Whether automated verdicts include compact per-card evidence. Set false to reduce response cost while retaining every label and confidence.
`seed`	0	Stable sampling seed.
`trust_reviewer`	true	Whether automated verdicts may influence suggestions. Set false to measure only.

Golden sets

Accepted human decisions form the golden set for the relevant task and Space. A curation card may include optional reasoning. That reasoning travels with the decision and helps future prompt suggestions understand whether the entire result was wrong or only one field was wrong. Read the golden-set and curation guide.

When suggestions run

Finishing a curation deck starts one improvement pass. If another deck finishes while it is running, the new evidence waits and may be combined into the next pass. Opening a report never starts improvement work. Read how suggestions become a successor version.

Admit, do not mutate

Suggestions do not edit a declared version. Create an editable successor, inspect each prompt or parameter suggestion, accept the ones you want, then declare the successor.