Efficient Decision-Making for Closed-Set Classification

    Closed-set decision tasks do not always need free-form generation. When the possible outcomes are predefined, models can score candidates directly in a single forward pass, returning labels, scores, and confidence instead of generating text for later parsing. This article explains the core task patterns, system architecture, calibration and evaluation practices, capacity planning, hybrid workflows with generative models, and the engineering checklist for reliable deployment.

    Why “Generate First, Parse Later” Is Often the Wrong Approach

    Many business workflows do not need an article or an explanation. They need an executable decision: whether an email is spam, which queue should receive a support ticket, whether an image satisfies moderation rules, or which document best matches a scoring rubric. Generative models are good at expressing answers in natural language, but this approach introduces avoidable overhead when the outcome space is closed.

    The first overhead is decoding. The model must generate tokens one by one, so latency increases with output length and throughput is constrained by the decoding stage. The second is parsing. Developers must extract “yes” or “no,” an option number, or a score from free-form text while handling paraphrases, unnecessary explanations, format drift, and occasional refusals. The third is uncertainty. A probability distribution over tokens is not the same as a posterior distribution over business options. Even when the text appears correct, it is difficult to use it directly as calibrated confidence for threshold-based policies.

    A more reliable abstraction is to provide the input, which may be multimodal, together with predefined questions and candidates. The model scores each candidate in one forward pass and directly returns labels, scores, and confidence. It does not invent answers outside the candidate set. This makes it particularly suitable for routing, quality inspection, pre-filtering before ranking, and agent delegation—cases where the business has already defined the decision space.

    Three Basic Forms of Decision Tasks

    In engineering systems, closed-set decisions can generally be grouped into three types. Keeping the interface semantics consistent makes monitoring and regression testing easier.

    Binary or yes/no classification. Typical examples include compliance detection, anomaly detection, and deciding whether human review is required. The output should include both the decision and the probability of the positive class. Business logic should use thresholds rather than hard labels: high-confidence cases can be processed automatically, low-confidence cases can be sent to human review, and the middle range can trigger additional rules or a more capable model. Thresholds must be selected on a validation set according to the costs of false positives and false negatives. The default value of 0.5 should not be adopted automatically.

    Finite-choice classification. The number of candidates should be constrained by the business rather than expanded without limits at runtime. Each option needs a stable identifier and clearly defined mutually exclusive or multi-select semantics. The model should return a distribution or ranking over options, not a sentence such as “I choose B because…”. If explanations are required, they should be implemented as an optional side capability rather than a mandatory part of the main path. Otherwise, the latency and parsing problems return.

    Rubric-based scoring. Scoring dimensions, ranges, and anchor examples should be included in the request contract instead of being hidden in a prose prompt. For example: “Completeness: 1–5, where 3 means that the main fields are covered but edge conditions are missing.” This makes scores comparable and suitable for regression testing. Whenever possible, score different dimensions separately so that unrelated defects are not mixed into one overall score.

    All three task types can be submitted in batches. A single request can contain multiple independent questions that share context encoding and return separate results. This is especially effective for checklists applied to the same document. However, the questions must remain semantically independent; a later question should not implicitly depend on an earlier model output.

    Where Single-Pass Scoring Fits in the System

    The processing pipeline can be divided into four layers:

    1. Contract layer: content, questions, candidates, scoring rubrics, and response fields. Candidate identifiers should use business keys rather than natural language as unique keys.
    2. Feature and encoding layer: text, images, audio, video, or documents are converted into a shared representation. Long documents should first be split structurally and associated with reference locations instead of sending every irrelevant page into one scoring request.
    3. Decision head: logits or regression scores are computed over the closed set and normalized into probabilities or calibrated scores. This step does not generate free-form text.
    4. Policy layer: thresholds, multi-label logic, human fallback, and audit logs. Model scores are not the same as final business actions.

    A vendor-neutral request might look like this:

    1. {
    2. "context": {"type": "document", "ref": "obj://reports/2024-q4"},
    3. "items": [
    4. {
    5. "id": "q1",
    6. "mode": "boolean",
    7. "question": "Does the document contain undisclosed related-party transaction language?",
    8. "positive_label": "yes"
    9. },
    10. {
    11. "id": "q2",
    12. "mode": "choice",
    13. "question": "Which processing queue should receive this document?",
    14. "options": [
    15. {"id": "finance", "text": "Finance review"},
    16. {"id": "legal", "text": "Legal review"},
    17. {"id": "ops", "text": "Operations handling"}
    18. ]
    19. }
    20. ]
    21. }

    The response should use stable fields such as label, score, confidence, candidates when a distribution is needed, model_version, and latency_ms. The model’s natural-language output should not be treated as the only source of truth.

    Calibration, Confidence, and Evaluation

    The quality of a closed-set decision system cannot be measured by accuracy alone. In highly imbalanced datasets, accuracy can hide poor recall. More useful metrics include:

    • Per-class precision, recall, and F1, together with business-cost-weighted error;
    • Macro and micro averages for multiclass tasks, so that large classes do not dominate the evaluation;
    • NDCG or pairwise agreement for ranking tasks;
    • Calibration error: whether predicted probabilities match observed frequencies. Reliable probabilities are what make threshold-based automation safe to scale.

    Calibration can be performed on a validation set using temperature scaling, isotonic regression, or bucket-based mapping. It must also be checked across data slices. Calibration curves may differ for different languages, document layouts, or support departments. Low confidence does not necessarily mean that the model is incapable; it is a policy signal. Collecting low-confidence cases into a labeling pool is often more effective than blindly increasing temperature or repeatedly modifying prompts.

    Evaluation sets should cover edge cases, including highly similar option descriptions, negation, contradictions between text and visuals in multimodal inputs, empty content, very long contexts, and shuffled candidate orders. Candidate order should be randomized during testing to detect stable position bias. Every production change should be evaluated against the same golden set. The comparison should examine distribution shift, not only a single accuracy number.

    Latency, Cost, and Capacity Planning

    A single forward pass without token-by-token decoding usually makes tail latency more predictable. The main cost depends on input length, modality, and the number of questions rather than on the length of a generated answer. Capacity planning should distinguish among:

    • Input side: document page count, image resolution, and audio or video duration usually dominate cost and latency;
    • Question side: multiple questions over the same context can share encoding, making their marginal cost lower than running a complete generation request for each question;
    • Output side: when the response contains only scores and labels, output billing and bandwidth are generally negligible, shifting the bottleneck back to encoding and the decision head.

    Hard limits are still necessary, including the maximum number of questions per request, option-text length, multimodal duration, concurrency, and queue timeout. Timeout behavior must specify whether the system fails fast or falls back to a rule engine. Unlimited retries can fill the queue. Oversized batches can also amplify tail latency, so batching should be designed around P95 rather than average latency.

    Cost comparisons must use the same task. If the baseline generates a long explanation and then extracts an answer with regular expressions, a decision interface will often be cheaper and more stable. If the task genuinely requires a readable explanation, plan, or code, the generative path remains necessary. Using the wrong abstraction merely to reduce cost can shift that cost into rework and human review.

    Working with Generative Models

    A practical system is usually hybrid. A decision model handles closed-set judgments, such as whether to call a tool, which specialized module to select, whether quality inspection passes, or which candidate answer best matches a rubric. A generative model handles open-ended output, such as writing explanations, summarizing differences, or producing a revised draft.

    One common orchestration pattern is to score multiple candidate drafts with a decision model and pass only those above a threshold to a generative model for expansion. Another is to have a generative model produce a limited set of candidates and then rerank them with a decision model using a fixed rubric. In the second pattern, the candidates must represent real alternatives. Otherwise, the generation step may introduce factual errors and the scoring model will merely select the best option from an already incorrect set.

    The same division works well for agent routing. Tool lists, permission scopes, and failure fallbacks should be defined by the application. Decision results should be written to audit logs, including a hash of the input summary, the option-set version, and the scores, so that the system can later explain why a particular branch was selected. Routing prompts should not be duplicated inconsistently across multiple services.

    Engineering Checklist for Production

    • Option governance: Version option text. Changing the option set changes the label space and requires reevaluation.
    • Data feedback loop: Feed low-confidence cases, human corrections, and production appeals into a shared labeling standard to prevent inconsistent definitions.
    • Security and privacy: Read documents and audio or video with least-privilege access. By default, logs should not store full content; retain only the references needed for traceability.
    • Observability: Monitor latency and score distributions by task type, modality, and content length. Alert on calibration drift, not only on error rates.
    • Fallbacks: If the model is unavailable, fall back to rules or the previous stable version and mark the result source. Never silently treat an unscored sample as approved.
    • Human review interface: Display scores and the triggered rubric points rather than only a color label. This makes review more efficient.

    When a platform provides shared model access, object storage, asynchronous queues, and monitoring, the decision service can operate as a stateless, horizontally scalable computation layer. Large files can remain in object storage, queues can absorb traffic spikes, and trace identifiers can associate the sub-items of a batch-scoring request.

    General cloud capabilities such as those provided by Baidu AI Cloud can support a structure in which inputs remain in storage, decisions are made online, and results are passed to a policy layer. Actual quotas, pricing, and API behavior depend on the services enabled and should be confirmed in the relevant service documentation. Demonstration figures should not be presented as capacity commitments.

    Conclusion

    The goal of closed-set decision-making is not to make models speak at greater length. It is to produce scores within a business-defined label space that can be calibrated, audited, and safely degraded. First determine which steps truly require generation and which only require judgment. Then define candidates, rubrics, thresholds, and human fallbacks as explicit contracts.

    This approach can substantially reduce latency and parsing failures while changing evaluation from “the answer looks correct” to “error is controlled on the samples that are appropriate for automation.” Open-ended creation should remain the responsibility of generative models. The two approaches should complement each other rather than replace one another.