LLM Selection for Production: Post-Training and Agent Orchestration

    This article explores practical strategies for selecting Large Language Models (LLMs) for production environments. It covers defining evaluation criteria, the impact of post-training, differences between minimal and standard invocation modes, calculating the cost per successful task, Agent orchestration best practices, and a decision matrix for model selection.

    1. Define Evaluation Criteria Before Judging Performance

    A single benchmark run cannot determine if a model is production-ready. Teams must at least record three things: the benchmark name and version, the baseline model for comparison, and the source and scale of task samples.

    If a software engineering benchmark score improves significantly after post-training, check if the benchmark questions overlap with the training data before retesting on a held-out private test set. Otherwise, the improvement might just be the result of benchmark contamination.

    2. The Role and Cost of Post-Training

    Pre-training determines how much knowledge a model stores, while post-training dictates how the model acts on instructions. The ratio of alignment data, instruction format, and tool-calling samples directly affect the final performance.

    If post-training overfits a specific execution framework, the model’s performance under other frameworks may degrade. During selection, test three types of tasks simultaneously: target scenarios, adjacent scenarios, and unseen general tasks. Generalization ability must be measured using a fixed test set, not just subjective impressions.

    3. Minimal Mode vs. Standard Mode

    Minimal mode exposes only a few core interfaces: input a prompt and output text. It consumes less context and has lower latency, making it suitable for tasks like summarization, classification, and rewriting.

    Standard mode registers a large number of tools. The model must determine whether to call a tool, which one to call, and what parameters to use. The more tool descriptions there are, the more context is consumed, and the higher the probability of selecting the wrong tool.

    Dimension Minimal Mode Standard Mode
    Number of Tools 0 Registered after intent filtering, usually just a few
    Context Usage Low Grows with the number of tool descriptions
    Suitable Tasks Summarization, classification, rewriting, sentiment analysis Multi-step queries, code generation, data aggregation
    Main Risks Cannot fetch external information Tool misselection, parameter errors, reasoning interference
    Testing Focus Output format and accuracy Tool selection accuracy, call success rate, end-to-end latency

    The pseudocode below assumes client is a wrapper for your SDK; the interface should be adjusted based on actual conditions.

    1. import time
    2. from dataclasses import dataclass
    3. @dataclass
    4. class ModelResult:
    5. text: str
    6. input_tokens: int
    7. output_tokens: int
    8. latency_ms: int
    9. def call_model_minimal(client, prompt: str, timeout_s: float = 30.0) -> ModelResult:
    10. """Minimal mode: only pass in the prompt, do not register any tools."""
    11. if not prompt.strip():
    12. raise ValueError("prompt cannot be empty")
    13. start = time.monotonic()
    14. resp = client.generate(prompt=prompt, timeout=timeout_s) # Raises TimeoutError on timeout
    15. latency = int((time.monotonic() - start) * 1000)
    16. return ModelResult(
    17. text=resp.text,
    18. input_tokens=resp.usage.input_tokens,
    19. output_tokens=resp.usage.output_tokens,
    20. latency_ms=latency,
    21. )
    1. TOOL_TOKEN_BUDGET = 1500 # Max context tokens allowed for tool descriptions, adjust based on the actual model
    2. def call_model_standard(client, prompt: str, all_tools: list[dict],
    3. max_tools: int = 8, timeout_s: float = 60.0) -> ModelResult:
    4. """Standard mode: filter tools by intent first, then register them to the model to avoid full registration."""
    5. selected = select_tools(prompt, all_tools, limit=max_tools) # Filtering done by routing or a small model
    6. if estimate_tokens(selected) > TOOL_TOKEN_BUDGET:
    7. raise ValueError("Tool descriptions exceed context budget, reduce tool count or simplify descriptions")
    8. start = time.monotonic()
    9. resp = client.generate(prompt=prompt, tools=selected, timeout=timeout_s)
    10. latency = int((time.monotonic() - start) * 1000)
    11. return ModelResult(
    12. text=resp.text,
    13. input_tokens=resp.usage.input_tokens,
    14. output_tokens=resp.usage.output_tokens,
    15. latency_ms=latency,
    16. )

    The key to standard mode is not having as many tools as possible, but exposing only the tools needed to complete the task for each request. The number of tools should be empirically determined: record tool selection accuracy and call success rate, find the inflection point where accuracy starts to drop, and set the upper limit just before that point.

    4. Parameter Scale and Cost Per Successful Task

    Comparing only single inference prices can lead to misjudgments. A more reasonable metric is the cost per successful task:

    Cost per successful task = (Total input token cost + Total output token cost + Failed retry cost) ÷ Number of successfully completed tasks

    The following figures are hypothetical and only for demonstrating the calculation. Assume a batch of 1,000 customer service tickets:

    • Large model: 930 successful, total cost $12, cost per successful task ≈ $0.0129.
    • Small model: 880 successful, total cost $3, cost per successful task ≈ $0.0034.

    The large model completed 50 more tasks at an extra cost of $9, averaging an extra $0.18 per additional task. If the business value of that task (e.g., reduced manual labor cost) exceeds $0.18, choosing the large model is more cost-effective; otherwise, choose the small model.

    Multi-model routing can be executed based on the following rules:

    • Trigger conditions: Route to the large model for complex reasoning, code generation, or long document analysis; route to the small model for simple Q&A, classification, or summarization.
    • Fallback strategy: Switch to the small model if the large model times out or gets rate-limited. Log the fallback event and mark the request for manual review.
    • Monitoring metrics: Cost per successful task, task success rate, P95 latency, retry rate, fallback rate, manual escalation rate. Check weekly and adjust routing thresholds promptly if cost or success rate deviates.

    5. Agent Orchestration Checklist

    Agent reliability depends primarily on the tool layer and orchestration layer design, not the model itself. Confirm the following before going live:

    1. Tool Definition: Describe each tool using JSON Schema, specifying parameter types, value ranges, and required fields. Example:
    1. {
    2. "name": "query_order",
    3. "description": "Query order status by order ID, read-only operation",
    4. "parameters": {
    5. "type": "object",
    6. "properties": {
    7. "order_id": {"type": "string", "pattern": "^[A-Z0-9]{12}$"}
    8. },
    9. "required": ["order_id"],
    10. "additionalProperties": false
    11. }
    12. }
    1. Timeout: Set individual timeouts for each tool and a total time budget for the entire task. Terminate and return a clear error message if the budget is exceeded.
    2. Retry: Automatically retry only idempotent operations using exponential backoff with random jitter, and set a maximum number of retries.
    3. Idempotency: Write operations must carry an idempotency key to prevent duplicate orders, deductions, or message sends.
    4. Permissions: Grant each tool only the minimum permissions required to complete the task. Register read-only tools separately from write tools.
    5. Manual Confirmation: Irreversible operations like deletion, sending, or payment must be confirmed by the user before execution.
    6. Observability: Log the tool name, desensitized parameter summary, latency, error code, and model version for each step to facilitate troubleshooting.

    6. Selection Decision Matrix

    Scenario Recommended Mode Model Size Key Metrics Main Risks
    Text summarization, sentiment analysis Minimal Mode Small Single call cost, format accuracy Missing key details
    Smart customer service, instruction execution Standard Mode (few tools) Small to Medium Cost per successful task Tool mis-invocation
    Code generation, data aggregation Standard Mode Large Failed retry cost, result accuracy Incorrect writes, benchmark contamination
    Long code refactoring, deep analysis Standard Mode + manual review Large Total latency, manual review workload Hallucinations, unsupported conclusions

    When selecting, determine the mode based on the scenario, calculate the cost per successful task using private samples, and finally decide whether routing is needed.

    7. Risk Warnings

    • Public benchmark scores may overlap with training data; retest using private samples before launch.
    • Prices and latency fluctuate with vendor adjustments; calculation results need periodic updates (quarterly reviews recommended).
    • When Agents call external services, duplicate writes or sensitive data leakage may occur. Tool permissions should be minimized, and sensitive fields in logs must be desensitized.
    • The code and figures in this article are illustrative; verify them with your specific SDK and business data before actual deployment.