Task workbench

Measure one semantic decision system with the deterministic steps that prepare it.

The Planner proposes the pipeline architecture. A task workbench is where you research one local decision system before trusting it: its semantic task, the normal operations or transformations that exist only to prepare that task, its measured quality, and its cost.

The Workbench button opens a compact index. Each group has its own focused page with a group-local dependency DAG, configuration, full-run estimates, sample controls, reviews, and result traces.

Below the DAG, the focused page uses two lanes. Task configuration shows a configured candidate gate as Python-equivalent code and lists every adjustable value with its task owner, such as cluster-outcomes.confidence_threshold. The right lane contains Cost estimate and Quality estimate. Cost covers the complete group on the configured full dataset. Quality uses the final semantic task's native metrics. Moving a control updates every available result from one configuration.

Create Review Sample and Run a Sample use a second two-lane row, in that order. Without a compatible sample, the cost estimate remains available and the left configuration lane explains which task must run before full-dataset quality can be estimated.

Suggested comparison values are finite settings worth testing, not the allowed domain and not evidence that those settings have already been evaluated. Empty gate and review panels are omitted.

An undeclared draft can be edited from the focused workbench. Saving operator parameters updates only registered task parameters. Saving a prompt publishes and binds a new immutable prompt revision. Adding or replacing a candidate gate accepts one typed, validated Python function plus any registered function-parameter schema. Declaration freezes these authoring controls.

Predecessor subtasks

A predecessor subtask is a deterministic operation or transformation whose sole purpose is to shape, contract, route, or enrich the hand consumed by the semantic task. For example, exact text cleanup and exact-duplicate contraction can sit beside a later semantic deduplication task in one workbench. Shared preprocessing that also feeds another consumer stays outside the group.

What an estimate means

A full-dataset cost estimate uses the configured data size, task shape, model pricing, and compatible completed execution samples. Random reviews of a completed full-dataset output estimate that output's quality. Reviews of a smaller execution do not establish the same quality for a larger run, whose decisions may change; its estimated range can therefore remain broad.

Supported confidence and registered gate-threshold changes can show cost and quality projections from compatible saved results and reviews. A projection is not a new execution. Its central value is a best-effort estimate, and its range retains uncertainty about new model answers and uncovered data. A quality range can span 0–100%; cost can have an upper bound that is not yet established. When a central value is unsupported, the range remains visible.

SemCluster quality is precision, recall, F1, and pair accuracy. SemTaxonomy and SemHierarchy quality is correct relation, wrong direction, and false independence. SemSort reports pairwise agreement and Kendall concordance. Other operators use their own terminal outcome metric.

Review metrics appear in the full-width Reviewed samples panel beneath the configuration and sample controls. A quality-estimate sample may update the full-dataset estimate and includes an interval. A balanced or researcher-selected sample is shown as a sample result only: it is useful for diagnosing errors but does not pretend to represent the full dataset.

Reviewed-sample history is durable across later prompt and configuration edits. Each batch states whether it is included in the current estimate. Earlier batches remain individually inspectable, but aggregate estimates use only evidence compatible with the current saved prompt and task shape.

For clustering, each labelled pair can identify whether a wrong result came from a predecessor premerge, a deterministic gate, the direct semantic judge, graph realization such as closure or repair, or incomplete execution. Rates from unrelated samples are never multiplied together.

SemCluster analysis can show predecessor, semantic-task-only, and composite quality separately. A deterministic premerge can make the composite score look healthy even when the semantic task contributes little, and a false predecessor merge cannot be repaired by the downstream cluster task.

Cost and gate research

The workbench shows estimated cost and quality. They use separate evidence. Cost uses compatible completed execution samples and never requires human labels. Quality uses representative reviewed results. Runtime is recorded only after execution. New compatible results update the estimates automatically. A candidate gate may additionally show relationships sent to the semantic model: relationships left for LLM judgment under the selected gate. This workload count is absent for parameters, such as confidence threshold, that do not control candidate routing. A run that stops before completing its sample does not qualify as a completed source for a full-dataset parameter-cost estimate.

Create a review sample

Use Run sample to selected step on the focused group page. Required earlier steps run in dependency order, and a registered external connector supplies a bounded sample to the first step without importing the full dataset.

A zero-row successful output is shown as depleted. Relifold does not secretly redraw until favorable rows survive. Increase the explicit source draw, choose a new seed, edit the responsible rule, or select a clearly labelled diagnostic stratum; rerunning an upstream task visibly invalidates its dependants.

After a step has a compatible research result, create a review sample from that result. A SemCluster full-dataset quality estimate first requires running the selected task over its full configured input. Relifold then draws random rows from that complete output. For each row, identify every other row that belongs in its true cluster. This measures both extra and missed matches. Five hundred rows is the default requested sample size, not a minimum for displaying estimates. Available reviewed rows contribute to the estimate, with ranges reflecting uncertainty. A partial execution or diagnostic sample remains available for error analysis and advice but does not claim full-dataset quality. If the first hand is still too wide, the estimate requests an evidence-sized additional hand; later hands exclude earlier anchors and reviewed rows across compatible hands contribute to the next estimate.

You may answer diagnostic cards manually, run Relifold's existing reviewer agent, or combine the two. Human answers always override reviewer answers. If you enable trust_reviewer, unresolved reviewer answers may provisionally drive workbench metrics. They remain visibly reviewer-sourced; they are not a human quality certificate. Reviewer assistance uses the terminal task's configured data model; the review API does not ask you to choose another provider or model. A SemCluster quality sample requires human identification of the complete matching set.

Under Reviewed samples, choose Run automated reviewer beside an eligible sample to request reviewer assistance. This uses paid model calls and refreshes the results when finished. Samples requiring complete matching sets explain why human review is needed. Automated answers, confidence scores, and reasons remain visible even when they are not trusted for estimates or advice; human-confirmed answers remain authoritative. Choose Open sample to open the same one-card-at-a-time interaction used for production curation in a new tab. Commit or skip each card and choose Finish deck when done. After pending saves finish, the workbench tab refreshes to show how many cards you have curated. Diagnostic verdicts can refresh improvement suggestions. Workbench verdicts remain local research evidence: they can support external-parameter and prompt suggestions but never become production priors. Compatible registered parameter settings are compared through the same optimizer used after a declared run; a suggestion appears only when replay supports it. Prompt suggestions use the same prompt-advisor contract and remain optional draft edits.

Explore registered parameters

The parameter laboratory reads the function's registered type, candidates, unit, and optional bounds. A bounded number uses a slider and exact input. Unbounded numbers use an exact input with an editable local viewport; that viewport is not a newly invented validity range. Candidate values are evaluated on the saved reviewed sample without changing the version.

A deterministic predicate replay can identify gate recall, independent-pair admission, and the number of relationships sent to the semantic model. Changing a SemCluster confidence threshold replays compatible saved decisions through the real fold, repair, and closure path. The page states when a new sample run is required because the setting would change which model calls exist.

Trace one result

The card trace API can expose a task-local path from displayed source values through predecessor subtasks, the producing task, final proposition, and reviewer/human answers. When persisted evidence supports attribution, error_source is upstream, a normal task id, or a semantic surface such as cluster-outcomes.gate, cluster-outcomes.judge, or cluster-outcomes.closure. Correct results omit error_source. When the source is *.judge, judge_observations lists every above-threshold direct observation for that pair. Each entry shows SAME or INDEPENDENT plus the exact group confidence and reason returned by the judge. For INDEPENDENT, left- and right-group assertions remain separate because the judge did not return one pair-local reason. Relifold never invents a reason or exposes the rest of the batch.

Python workflow

groups = workspace.list_workbench_groups("event-recommendations", "v1")

workspace.save_workbench_group(
    "event-recommendations",
    "v1",
    workbench_task_id="event-dedup",
    name="Event deduplication",
    semantic_task_id="deduplicate-events",
    subtask_ids=["clean-headlines", "deduplicate-events"],
    predecessor_task_ids=["clean-headlines"],
)

group = workspace.workbench_group("event-recommendations", "v1", "event-dedup")
configuration = group.inspect()

estimate = group.fetch_estimate({
    "deduplicate-events.confidence_threshold": 0.5,
})
cost = estimate["cost_estimate"]
quality = estimate["quality_estimate"]

advance = group.run_sample(
    target_task_id="deduplicate-events",
    sample_size=100,
    sample_seed=7,
    maximum_cost_usd=2.00,
)

# Reviews are separate: use balanced cards for diagnosis and advice.
review = group.create_review(
    cards=30,
    seed=7,
    selection="balanced",
    trust_reviewer=True,
)

review_id = review["review_id"]
group.run_reviewer(review_id, maximum_cost_usd=1.00)

group.submit_verdict(
    review_id,
    card_id="one-card-id",
    labels={"same": True, "independent": False},
    reason="Both headlines report the same concrete event.",
)

quality_review = group.create_review(cards=200, seed=8, selection="random")
group.submit_verdict(
    quality_review["review_id"],
    card_id="one-anchor-card-id",
    matching_keys=["event-102", "event-377"],
)

review_ids = group.list_reviews()
one_result = group.fetch_review_results(review_id)
all_results = group.fetch_review_results()
advice = group.refresh_advice(review_id, maximum_cost_usd=10.00)
trace = group.trace(review_id, "one-card-id")

If a sample was created without trusting automated reviews, use group.use_reviewer_answers(review_id) to include its saved answers without more model calls. Human corrections take precedence. The browser uses automated answers when you run its reviewer. Diagnostic review results describe the reviewed cards; they do not establish full-dataset quality.

The browser exposes the same surface from a version's Workbench button. Nothing silently modifies a prompt, predicate, parameter, or DAG.