SemCluster

Recover groups of equivalent or criterion-related values within one Space.

SemCluster is the semantic deduplication operation. It is also useful for grouping mentions of the same event, entity, intent, or concept. The prompt defines exactly what may share a cluster.

from scratchpad.api import SemCluster, gates

SemCluster(
    name="group-news-events",
    input_spaces=("headlines",),
    output_space="event-clusters",
    primary_key="headline",
    prompt=(
        "Group headlines only when they report the same developing event. "
        "A shared person, company, sport, or broad theme is not enough."
    ),
    provider="openai",
    model="gpt-5",
    reasoning_effort="low",
    batch_size=16,
    rho=0.30,
    merge_only_if=gates.equal("category"),
)

Deduplication boundary

Do not merge a more specific value into a broader one merely because the broader value summarizes it. “Workout” may match a true synonym such as “exercise,” while “running” and “camping” retain information that “workout” does not.

Exact duplicates

If equality is a column comparison, use Union(key=...). It is deterministic and does not spend model budget.

Indexed hard constraints

Use merge_only_if=gates.equal("category") when different categories are a true final incompatibility. For human-entered text, gates.equal("category", normalize_text=True, require_nonempty=True) compares lowercase, whitespace-normalized values while rejecting blanks. The predicate is both the indexed candidate boundary and a permanent cannot-link rule: any rejected pair can never merge.

Do not place approximate retrieval, lexical similarity, or embedding thresholds in merge_only_if unless their false pairs are genuinely impossible under the authored clustering definition. Keep a lossy approximation in an explicit predecessor task so its recall tradeoff remains visible and reviewable.

Calibrated gate parameters

from scratchpad.api import FunctionParameter, PythonFunction, SemCluster, gates

eligible_event_pair = PythonFunction(
    name="eligible_event_pair",
    source="""def eligible_event_pair(left, right, *, max_days):
    if left.get('category') != right.get('category'):
        return False
    return abs(left['event_day'] - right['event_day']) <= max_days
""",
    parameters={
        "max_days": FunctionParameter(
            value_type="number",
            default=90,
            minimum=1,
            maximum=365,
            candidates=(30, 90, 180),
            description="Maximum possible span of one event thread.",
        ),
    },
    candidate_gate=gates.equal("category", require_nonempty=True),
)

SemCluster(
    name="group-events",
    input_spaces=("event-signatures",),
    output_space="events",
    primary_key="event_signature",
    evidence=("title", "event_day"),
    prompt="Merge only reports of the same bounded real-world occurrence.",
    provider="openai",
    model="gpt-5",
    reasoning_effort="low",
    merge_only_if=eligible_event_pair,
    function_parameters={"max_days": 90},
)

The registered function owns the parameter schema and finite candidates; the task owns this version's resolved value. The candidate index must be a complete sparse-enumeration precondition, while the function applies the full hard boundary. Parameter advice remains unavailable unless the review hand covers every pair a candidate setting could expose.

Large and growing catalogs

Do not send an unbounded connector Space directly into SemCluster. First use a linear SemTransform to produce a compact meaning-based canonical signature, consolidate exact signatures while retaining every source identifier, and bound the representative hand before semantic refinement. Deterministic token cleanup or lexical signatures can assist retrieval but do not replace the semantic assignment. Keep the full identifier mapping on a separate lookup path and overlay bounded refinements rather than dropping unselected assignments. A call or dollar cap limits spend; it does not turn an unbounded relationship result into a complete partition.

Cost estimate

SemCluster forecasting is based on the final quotient structure. For N values, M expected final clusters, and Q still-eligible pairs between those clusters, the certificate has (N-M)+Q witnesses. Compatible complete executions and draft-research hands teach cluster growth, quotient density, and physical packing. One completed hand calibrates packing at its exact extent; cross-extent packing requires at least three distinct completed extents. With less evidence, perfect packing remains a labelled lower-bound scenario and expected cost stays unknown. Relifold then recomputes the operator geometry for the current extent; it never multiplies a smaller run's cost by the row ratio. The UI reports the mathematical floor, evidence-backed expected interval, and candidate-retirement ceiling separately. Cold token pricing uses the operator's actual response shape: node-keyed partition hands for SemCluster, rather than pair-label volume. A compatible complete exact-shape hand supersedes that structural envelope.