Golden sets and human curation

Record reusable human truth with optional reasoning.

A golden set is the accepted human truth established by curation. It belongs to the reviewed task and output meaning, so the same reviewed witness may measure every compatible task that produced it.

Create a curation deck

Set curation_sample_size to the number of independent cards you want. A positive number enables curation; there is no second enable switch and no directory to configure.

An unfinished deck is stable: reopening or exporting it returns the same cards. Finishing the deck retires that hand, so the next deliberate open can draw another sample that excludes accepted human truth.

Review(
    name="event-thread-quality",
    tasks=("group-events",),
    sample_size=100,
    curation_sample_size=30,
    provider="openai",
    model="gpt-5",
    reasoning_effort="low",
    trust_reviewer=False,
)

Grade a card

Mark the claim good or bad and optionally explain the correction. For a structured output, reasoning can identify the field that is wrong without rejecting the parts that are correct—for example, “yield sentiment should be +1; slope sentiment is correct.”

The UI saves one deck together. API clients can do the same with workspace.curate_many(...); each verdict supplies the untouched card, a complete label-answer mapping, and optional reasoning. The server validates the entire request against the checkpoint before saving any card.

deck = workspace.curation_deck(
    "news-recommendations", "v2", "event-thread-quality"
)
verdicts = [
    {
        "claim": card,
        "labels": {"same": True, "independent": False},
        "reason": "Both headlines report the same launch event.",
    }
    for card in deck["claims"]
]
workspace.curate_many(
    "news-recommendations",
    "v2",
    "event-thread-quality",
    verdicts=verdicts,
)

What the golden set affects

Golden truth contributes to task quality metrics and future suggestions. It does not rewrite a historical output row, erase another session's result, or automatically mutate a declared version. A relationship or keep/drop decision may also apply to compatible outputs produced in future sessions. A transformation decision applies only to the exact produced value that was reviewed: repeating that value reuses the decision, while changing it creates a new card.

Inspect it

Open the checkpoint or a covered task in its pipeline version. The golden set, reviewer metrics, and task report are available there and through the workspace API.