LLM Selection: How to Evaluate Coding, Long-Context, and Private Deployment

    Choosing an LLM requires task-specific evaluation rather than leaderboard ranking alone. This article outlines practical methods for assessing coding ability, long-context comprehension, multimodal inputs, agent orchestration, latency, concurrency, cost, and deployment/compliance constraints, with example measurement code and scoring checklists.

    LLM Selection: How to Evaluate Coding, Long-Context, and Private Deployment

    1. Answer Three Questions Before Selection

    Many teams start by checking public leaderboards and picking the highest-scoring model. This approach has two problems. First, leaderboard test items often differ substantially from a team’s real business tasks. Second, a single score cannot reflect practical constraints such as latency, cost, and deployment requirements.

    A more reliable approach is to answer three questions first:

    • What type of task is this, and what are the input and output forms?
    • What are the tolerances for correctness, response speed, and cost?
    • Can the data leave the intranet, and are there compliance restrictions?

    The answers to these questions become the evaluation criteria. Without clear criteria, test results cannot be judged as good or bad.

    2. Break the Business Down into Evaluable Tasks

    The first step in model selection is to decompose the business into several task types and evaluate each one separately. Common task types include:

    • R&D assistance: code generation, defect localization, code review.
    • Knowledge processing: long-document reading, meeting-notes analysis, cross-comparison of multiple sources.
    • Multimodal processing: screenshot understanding, converting UI design mockups into frontend code, information extraction from scanned documents.
    • Automation workflows: multi-tool calls, long-chain tasks, batch execution.
    • User-facing services: conversation quality, response speed, concurrency capacity during peak hours.

    Each task type places different demands on model capabilities. A model that performs well in code review may not be suitable for handling user inquiries at scale. Evaluation should therefore be performed by task and then aggregated.

    3. Code Generation and Engineering Capability

    When evaluating coding ability, cover three scenarios:

    1. Implement a function containing business rules based on a requirement description.
    2. Provide an architecture proposal for a module and explain the trade-offs of each option.
    3. Review a change and identify logic flaws, missing edge cases, and security issues.

    Scoring should not depend only on whether the code runs. It should also check whether the model follows the team’s coding standards, covers exception branches, and accurately explains the business logic. Below is a simple review scoring example:

    1. REVIEW_CHECKLIST = [
    2. "Are inputs validated at the trust boundary",
    3. "Are exception branches handled explicitly",
    4. "Are there race conditions in concurrent access",
    5. "Are sensitive fields leaked in logs",
    6. "Is there a corresponding test case for new logic",
    7. ]
    8. def score_review(model_findings, known_issues):
    9. """Calculate the share of known issues found by the model and record false positives."""
    10. hit = sum(1 for issue in known_issues if issue in model_findings)
    11. false_pos = sum(1 for f in model_findings if f not in known_issues)
    12. return {
    13. "recall": hit / len(known_issues),
    14. "false_positives": false_pos,
    15. }

    Recall reflects whether the model misses issues, while the number of false positives reflects whether it creates noise. Both metrics matter. A review tool with too many false positives consumes engineers’ attention and is eventually ignored by the team.

    4. Long-Context Understanding

    A nominal context window length does not equal effective understanding capability. The longer the text, the more likely the model is to miss middle sections or mix details from different paragraphs.

    Two methods can be used to test this:

    • Needle test: insert several verifiable facts at different positions in a long document, then ask the model questions. Measure recall and accuracy, and observe whether accuracy drops significantly with position or text length.
    • Cross-document task: provide three reports and ask the model to compare the same metric across all three and explain the evidence. This type of task tests cross-reasoning and reflects reading depth better than simple retrieval.
    1. def build_long_context(docs, facts):
    2. """Insert several facts at random positions in the documents and return the concatenated context and reference answers."""
    3. context = "\n".join(docs)
    4. answers = {}
    5. for key, (sentence, question, answer) in facts.items():
    6. context = insert_at_random_position(context, sentence)
    7. answers[key] = (question, answer)
    8. return context, answers

    Tests should fix context-length gradients, such as 8K, 32K, and 128K, and record accuracy curves separately. The inflection point of the curve often says more than nominal parameters.

    5. Multimodal Inputs

    Two points are key to multimodal capability: whether input types are natively supported and what the understanding quality is after support.

    Items to confirm include:

    • Whether screenshots, PDFs, and design mockups can be input directly, or whether OCR or format conversion is required first. Conversion steps add errors and maintenance cost.
    • Video usually requires frame sampling. The number of sampled frames directly affects cost and latency, so evaluation should record results under different sampling counts.

    Evaluation metrics can include layout information extraction accuracy, chart value reading error, and UI element restoration fidelity. For design-to-code tasks, also check whether the generated code renders correctly in the target framework.

    6. Agents and Multi-Tool Orchestration

    The difficulty of long-chain tasks lies in controllability. A model may fall into repeated calls or pass incorrect arguments. Engineering should set explicit boundaries for agents:

    1. MAX_STEPS = 12
    2. def run_agent(task, tools, model):
    3. history = [{"role": "user", "content": task}]
    4. for _ in range(MAX_STEPS):
    5. reply = model.generate(history, tools=tools)
    6. if reply.is_final:
    7. return reply.text
    8. if reply.tool_name not in tools:
    9. raise ValueError("Model called an unauthorized tool")
    10. validate_arguments(reply.tool_name, reply.arguments) # argument allowlist validation
    11. result = tools[reply.tool_name](**reply.arguments)
    12. history.append({"role": "tool", "content": result})
    13. raise RuntimeError("Maximum steps exceeded; task terminated")

    This code reflects three principles: limit the number of steps to prevent infinite loops; validate tool names and arguments to prevent unauthorized or incorrect calls; and raise failures explicitly so upper layers can retry or involve human operators.

    When evaluating agents, track task completion rate, average number of tool calls, invalid argument ratio, and the proportion of failures caused by exhausting the step limit.

    7. Inference Speed, Concurrency, and Cost

    Speed and cost are among the easiest metrics to oversimplify. Common definitions are:

    • Time to first token (TTFT): the time from sending a request to receiving the first output chunk, which determines perceived responsiveness.
    • Output throughput: the number of tokens generated per second, which determines total time for long-text generation.
    • Concurrency capacity: the number of requests that can be served stably within the target latency, which determines whether queueing occurs during peak hours.
    • Cost per task: total input and output tokens for a complete task multiplied by unit price.

    Two points require attention. First, high throughput does not necessarily mean a good experience. Latency jitter under high concurrency has a greater impact on users, so tail latency such as P95 and P99 should be examined. Second, token-to-character conversion depends on the tokenizer, so cost cannot be estimated simply by character count.

    1. import time
    2. def measure(stream_fn, prompts):
    3. """Approximate TTFT and output speed by counting streamed chunks."""
    4. results = []
    5. for prompt in prompts:
    6. start = time.perf_counter()
    7. first_chunk_at = None
    8. chunk_count = 0
    9. for _ in stream_fn(prompt):
    10. if first_chunk_at is None:
    11. first_chunk_at = time.perf_counter() - start
    12. chunk_count += 1
    13. total = time.perf_counter() - start
    14. results.append({
    15. "ttft": first_chunk_at,
    16. "chunks_per_sec": chunk_count / total,
    17. })
    18. return results

    This code counts returned chunks rather than exact tokens. For exact results, use the counting method provided by the model service provider.

    8. Deployment Models and Data Compliance

    There are three common deployment models: calling public cloud service APIs, deploying in a private cloud or private data center, and using a hybrid of both.

    The choice of deployment model should be determined by data classification. Confidential or internally sensitive materials can usually be processed only in an intranet environment. In such cases, the security team should first confirm the permitted deployment form before discussing model quality.

    Items to verify include:

    • Whether data will be used for model training and what the service provider’s terms specify.
    • Log retention duration and access auditing.