Task workbench

Measure one semantic decision system with the deterministic steps that prepare it.

The Planner proposes the pipeline architecture. A task workbench is where you research one local decision system before trusting it: its semantic task, the normal operations or transformations that exist only to prepare that task, its measured quality, and its cost.

The Workbench button opens a compact index. Each group has its own focused page with a group-local dependency DAG, configuration, full-run estimates, sample controls, reviews, and result traces.

Below the DAG, the focused page uses two lanes. Task configuration shows a configured candidate gate as Python-equivalent code and lists every adjustable value with its task owner, such as cluster-outcomes.confidence_threshold. The right lane contains Cost estimate and Quality estimate. Cost covers the complete group on the configured full dataset. Quality uses the final semantic task's native metrics. Moving a control updates every available result from one configuration.

Create Review Sample and Run a Sample use a second two-lane row, in that order. Without a compatible sample, the cost estimate remains available and the left configuration lane explains which task must run before full-dataset quality can be estimated.

Suggested comparison values are finite settings worth testing, not the allowed domain and not evidence that those settings have already been evaluated. Empty gate and review panels are omitted.

An undeclared draft can be edited from the focused workbench. Saving operator parameters updates only registered task parameters. Saving a prompt publishes and binds a new immutable prompt revision. Adding or replacing a candidate gate accepts one typed, validated Python function plus any registered function-parameter schema. Declaration freezes these authoring controls.

Predecessor subtasks

A predecessor subtask is a deterministic operation or transformation whose sole purpose is to shape, contract, route, or enrich the hand consumed by the semantic task. For example, exact text cleanup and exact-duplicate contraction can sit beside a later semantic deduplication task in one workbench. Shared preprocessing that also feeds another consumer stays outside the group.

What an estimate means

A full-dataset cost estimate uses the configured data size, task shape, model pricing, and compatible completed execution samples. Random reviews of a completed full-dataset output estimate that output's quality. Reviews of a smaller execution do not establish the same quality for a larger run, whose decisions may change; its estimated range can therefore remain broad.

Supported confidence and registered gate-threshold changes can show cost and quality projections from compatible saved results and reviews. A projection is not a new execution. Its central value is a best-effort estimate, and its range retains uncertainty about new model answers and uncovered data. A quality range can span 0–100%. A point and a statistical interval appear together; limited evidence can make the interval broad. Both are unavailable when required inputs are missing.

SemCluster quality is precision, recall, F1, and pair accuracy. SemTaxonomy and SemHierarchy quality is correct relation, wrong direction, and false independence. SemSort reports pairwise agreement and Kendall concordance. Other operators use their own terminal outcome metric.

Review metrics appear in the full-width Reviewed samples panel beneath the configuration and sample controls. These are exact measurements of the reviewed cards, without intervals. The separate Quality estimate panel projects to the full dataset and shows a point and an interval together. Unrepresentative or limited evidence can produce broad intervals; a sample result does not by itself establish full-dataset quality.

Reviewed-sample history is durable across later prompt and configuration edits. Each batch states whether it is included in the current estimate. Earlier batches remain individually inspectable, but aggregate estimates use only evidence compatible with the current saved prompt and task shape.

For clustering, each labelled pair can identify whether a wrong result came from a predecessor premerge, a deterministic gate, the direct semantic judge, graph realization such as closure or repair, or incomplete execution. Rates from unrelated samples are never multiplied together.

SemCluster analysis can show predecessor, semantic-task-only, and composite quality separately. A deterministic premerge can make the composite score look healthy even when the semantic task contributes little, and a false predecessor merge cannot be repaired by the downstream cluster task.

Cost and gate research

The workbench shows estimated cost and quality. They use separate evidence. Cost uses compatible completed execution samples and never requires human labels. Quality uses representative reviewed results. Runtime is recorded only after execution. New compatible results update the estimates automatically. A candidate gate may additionally show relationships sent to the semantic model: relationships left for LLM judgment under the selected gate. This workload count is absent for parameters, such as confidence threshold, that do not control candidate routing. A run that stops before completing its sample does not qualify as a completed source for a full-dataset parameter-cost estimate.

Estimate calls may take several minutes. The Python client waits automatically; an explicitly configured timeout_seconds still applies. The web panel shows Estimating… while the selected configuration is being evaluated.

Create a review sample

Use Run sample to selected step on the focused group page. Required earlier steps run in dependency order, and a registered external connector supplies a bounded sample to the first step without importing the full dataset.

A zero-row successful output is shown as depleted. Relifold does not secretly redraw until favorable rows survive. Increase the explicit source draw, choose a new seed, edit the responsible rule, or select a clearly labelled diagnostic stratum; rerunning an upstream task visibly invalidates its dependants.

After a step completes a research sample, create reviews from that result. Review enough varied examples to represent the task, including both correct and incorrect outcomes. For clustering, look for both extra matches and missed matches. Available reviewed evidence contributes to the estimate; there is no minimum review count for displaying a point and interval. An unfinished execution remains useful for diagnosis but is not a completed estimation sample. New compatible sample executions and reviews contribute to later full-dataset estimates.

You may answer diagnostic cards manually, run Relifold's existing reviewer agent, or combine the two. Human answers always override reviewer answers. If you enable trust_reviewer, unresolved reviewer answers may provisionally drive workbench metrics. They remain visibly reviewer-sourced; they are not a human quality certificate. Automated review defaults to GPT-5.6 Sol with high reasoning effort, independently of the worker model. Choose another reviewer using the browser's model controls or the Python client's provider, model, and reasoning_effort arguments. A SemCluster quality sample requires human identification of the complete matching set.

Under Reviewed samples, choose Run automated reviewer beside an eligible sample to request reviewer assistance. This uses paid model calls and refreshes the results when finished. Samples requiring complete matching sets explain why human review is needed. Automated answers, confidence scores, and reasons remain visible even when they are not trusted for estimates or advice; human-confirmed answers remain authoritative. Choose Open sample to open the same one-card-at-a-time interaction used for production curation in a new tab. Commit or skip each card and choose Finish deck when done. After pending saves finish, the workbench tab refreshes to show how many cards you have curated. Diagnostic verdicts can refresh improvement suggestions. Workbench verdicts remain local research evidence: they can support external-parameter and prompt suggestions but never become production priors. Compatible registered parameter settings are compared through the same optimizer used after a declared run; a suggestion appears only when replay supports it. Prompt suggestions use the same prompt-advisor contract and remain optional draft edits.

Explore registered parameters

The parameter laboratory reads the function's registered type, candidates, unit, and optional bounds. A bounded number uses a slider and exact input. Unbounded numbers use an exact input with an editable local viewport; that viewport is not a newly invented validity range. Candidate values are evaluated on the saved reviewed sample without changing the version.

A deterministic predicate replay can identify gate recall, independent-pair admission, and the number of relationships sent to the semantic model. Changing a SemCluster confidence threshold replays compatible saved decisions through the real fold, repair, and closure path. The page states when a new sample run is required because the setting would change which model calls exist.

Trace one result

The card trace API can expose a task-local path from displayed source values through predecessor subtasks, the producing task, final proposition, and reviewer/human answers. When persisted evidence supports attribution, error_source is upstream, a normal task id, or a semantic surface such as cluster-outcomes.gate, cluster-outcomes.judge, or cluster-outcomes.closure. Correct results omit error_source. When the source is *.judge, judge_observations lists every above-threshold direct observation for that pair. Each entry shows SAME or INDEPENDENT plus the exact group confidence and reason returned by the judge. For INDEPENDENT, left- and right-group assertions remain separate because the judge did not return one pair-local reason. Relifold never invents a reason or exposes the rest of the batch.

Python workflow

groups = workspace.list_workbench_groups("event-recommendations", "v1")

group = workspace.workbench_group(
    "event-recommendations", "v1", "event-dedup",
    tasks=["clean-headlines", "deduplicate-events"],
    name="Event deduplication",
)
configuration = group.inspect()

estimate = group.fetch_estimate({
    "deduplicate-events.confidence_threshold": 0.5,
})
cost = estimate["cost_estimate"]
quality = estimate["quality_estimate"]

sample = group.run_sample(
    target_task_id="deduplicate-events",
    sample_size=100,
    sample_seed=7,
    max_cost_usd=2.00,
)
# Continue with review only after the sample is complete.

# Reviews are separate: use balanced cards for diagnosis and advice. A review whose reviewer
# answers count as truth starts that reviewer in the background; fetch_estimate waits for it.
review = group.create_review(
    cards=30,
    seed=7,
    selection="balanced",
    trust_reviewer=True,
)
review_id = review["review_id"]

group.submit_verdict(
    review_id,
    card_id="one-card-id",
    labels={"same": True, "independent": False},
    reason="Both headlines report the same concrete event.",
)

quality_review = group.create_review(cards=200, seed=8, selection="random")
group.submit_verdict(
    quality_review["review_id"],
    card_id="one-anchor-card-id",
    matching_keys=["event-102", "event-377"],
)

review_ids = group.list_reviews()
one_result = group.fetch_review_results(review_id)
all_results = group.fetch_review_results()
advice = group.refresh_advice(review_id, max_cost_usd=10.00)
trace = group.trace(review_id, "one-card-id")

To have the reviewer answer a review created without trust_reviewer, or to use another model, call group.run_reviewer(review_id, provider=..., model=...). Human corrections take precedence. The browser uses automated answers when you run its reviewer. Diagnostic review results describe the reviewed cards; they do not establish full-dataset quality.

The browser exposes the same surface from a version's Workbench button. Nothing silently modifies a prompt, predicate, parameter, or DAG.