Open-Source LLM Selection and Agent Engineering Adaptation Guide
This guide explains how to select open-source large language models for production and adapt them for agent engineering. It covers license and weight verification, business-specific evaluation sets, JSON Schema-constrained tool calling, plan-based agent execution, deployment and heterogeneous hardware adaptation, quantization and batch testing, layered serving, and the structure of a controlled execution loop. Readers get a practical checklist and implementation patterns for reducing model-selection risk and improving agent reliability.
1. Verify Licenses and Weights First
New versions of open-source models appear quickly, but whether a model can enter production first depends on whether several facts are clear. Teams should complete the checks in the table below before evaluation and store the results in an internal compliance checklist.
| Check item | What to record | How to handle it |
|---|---|---|
| Weight source | Official repository URL, version tag, file SHA256 | Compare the hash after download; stop using the file if it does not match |
| License type | License name and version number | Ask legal to confirm whether commercial use is allowed |
| Attribution and derivative terms | Whether attribution is required, whether derivative models must be released under the same license | Affects whether fine-tuned models can be distributed externally |
| Usage restrictions | Prohibited scenarios, user limits, or revenue caps | Compare each item against the product scenario |
| Context length | Maximum token count stated officially | Use measured results as the basis; do not directly adopt marketing values |
| Inference framework support | Supported inference frameworks and versions | Determines the later deployment path and operator coverage |
Weight availability also determines whether private deployment, offline inference, and further fine-tuning are feasible. If weights require calling a third-party API online, evaluate separately whether data leaves your environment.
2. Use Business Evaluation Sets Instead of Leaderboard Scores
Leaderboard scores can only be used for initial screening. Prompt templates, decoding parameters, and data versions on leaderboards often differ from business scenarios, and the results are hard to reproduce. Teams should build their own business evaluation set and meet the following requirements:
- Stratify by task type, covering question answering, summarization, structured extraction, code understanding, multi-step reasoning, and tool calling. Each category should have no fewer than 50 samples, and the total should be no fewer than 200.
- Give the dataset a version number, and record sample changes in a changelog.
- Fix the prompt template, temperature, top_p, and random seed, and record the inference framework version.
- Version the scoring scripts separately. Use scripts for tasks that can be judged by rules; for open-ended tasks, use double-blind review by two people, with a third person adjudicating disagreements.
- Rerun every model upgrade on the same dataset, and save the results side by side with historical versions.
Demo videos and single impressive outputs cannot be used as the basis for model selection. The selection report should include the pass rate for each task type under the same conditions.
3. Tool Calling: Constrain Parameters with JSON Schema
Tool-calling failures usually come from invalid parameters, unclear descriptions, and truncated return results. Tool definitions should use structured descriptions, including name, purpose, parameter types, required fields, and prohibited additional fields. Below is an example definition of a read-only query tool:
{"name": "query_order","description": "Query order status by order number. Read-only operation; does not modify any data","parameters": {"type": "object","properties": {"order_id": {"type": "string","pattern": "^ORD[0-9]{10}$","description": "Order number in the format ORD followed by 10 digits"}},"required": ["order_id"],"additionalProperties": false}}
After the orchestrator receives a tool-call request from the model output, it should perform the following checks in order:
- Is the tool name in the whitelist for the current task?
- Do the parameters pass JSON Schema validation?
- Does the write operation have an idempotency key and an approval record?
- After the tool returns, keep only the fields relevant to the task before writing them back into the context.
Error codes should be defined consistently to support statistics and troubleshooting:
| Error code | Meaning | Handling strategy |
|---|---|---|
| E_TOOL_NOT_ALLOWED | Tool is not in the whitelist | Log it and let the model choose again |
| E_PARAM_MISSING | Required parameter is missing | Ask the model to request clarification from the user |
| E_PARAM_INVALID | Parameter type or format is invalid | Write the validation information back and retry at most once |
| E_TIMEOUT | Tool timed out | Read-only tools can be retried; write tools are handed off to a human |
| E_UPSTREAM_ERROR | Upstream service returned an error | Log the full response and handle it according to the fallback strategy |
For compatibility, most inference services support similar function calling fields, namely name, description, and parameters in JSON Schema form. However, different models and service frameworks vary greatly in how closely they follow these fields. Before integration, construct a set of tool-call samples and measure the proportion of model outputs that can be parsed and pass Schema validation.
4. Agent Execution Loop: Plan Structure and Control Parameters
Before execution, the model first generates a plan. The plan should have a fixed structure so the orchestrator can check and record it. Below is an example plan:
{"plan_id": "plan-20261119-001","task_id": "t-001","max_steps": 8,"budget_seconds": 60,"steps": [{"step_id": "s1","goal": "Query status based on the order number provided by the user","tool": "query_order","depends_on": [],"success_condition": "The returned result contains the status field"},{"step_id": "s2","goal": "Generate a reply based on the order status","tool": null,"depends_on": ["s1"],"success_condition": "The reply contains the order status and no internal fields"}]}
Control parameters should be set according to task risk. For general read-only tasks, set 5 to 8 steps and a 60-second budget. For tasks involving write operations, reduce the step count and add a human confirmation checkpoint before the write operation.
5. Deployment and Heterogeneous Adaptation
Deployment adaptation should be based on measured data. The matrix below can be used as a template to fill in; values need to be measured on the target hardware.
| Dimension | What to record | Example acceptance criterion |
|---|---|---|
| Card type and driver | Accelerator card model, driver, and runtime version | Consistent with the official support list of the inference framework |
| Operator coverage | Whether all operators used by the model are implemented | No warning at startup about fallback to a slow path |
| Parallel strategy | Configuration of tensor parallelism or pipeline parallelism | GPU memory headroom not less than 15% at target concurrency |
| Quantization method | Precision, calibration data, quantization scope | Structured output compliance rate drops by no more than 2 percentage points |
| Throughput and latency | Time to first token, P95 latency, requests per second | Meets the business SLA, for example P95 below 3 seconds |
| Serving | Container image, scaling rules, health checks | Can take traffic within 2 minutes after scaling |
Quantization requires joint evaluation. Quantization reduces GPU memory usage, but it may cause errors in structured output and tool parameters. Before and after quantization, compare compliance rates on the same structured-output test set and separately measure tool-call parameter accuracy.
Batch processing should be tested at target concurrency. Latency for a single request cannot represent service capacity. Stress test with a real request distribution and record P50, P95, and timeout rate.
Layered serving can also reduce costs. Simple questions can be handled by rules or small models, while complex tasks go to larger models and agent orchestration. Fixed knowledge queries and tool return results can be cached. Cache keys must include the tool version and parameter hash to avoid misuse of stale results.
6. A Runnable Execution Loop Skeleton
The code below implements a controlled execution loop, including whitelist checks, parameter validation, timeout control, no retries for write operations, and a step budget. The model call is injected through the decide function.
