Large Language Model Selection Guide: Technical Trade-offs in Capability, Deployment, and Cost
Choosing an LLM is not about picking the highest-scoring model, but the one that fits your constraints. This guide breaks the decision into four technical dimensions—capability, deployment, performance, and cost—and explains how to reason from business requirements to verifiable metrics. It covers native vs. bolted-together multimodal handling, text reasoning and instruction following, long-context retrieval, public-cloud API vs. private deployment and the role of open-weight models, throughput and latency (including Little's Law for capacity planning), full cost-of-ownership modeling, and scenario-based strategies for high-concurrency consumer apps and batch/agent workloads, ending with a practical go-live checklist.
1. Selection Is Not About “Which Is Stronger,” but “Which Fits Better”
The most common mistake in LLM selection is reducing it to “higher benchmark score is better.” Real business decisions are far more complex than a leaderboard: a high-concurrency, consumer-facing application and a back-office system that handles sensitive internal documents have almost opposite requirements. The former cares about throughput, latency, and unit price; the latter cares about data sovereignty and the feasibility of private deployment.
So the right approach is to work backward from business constraints to technical requirements, then use quantifiable metrics to verify whether candidate solutions meet the bar. Below, the evaluation is broken into four dimensions: capability, deployment, performance, and cost.
2. Capability: First, Define What the Business Actually Needs
2.1 Multimodal Capability: Native Support vs. Assembled Pipelines
Many use cases require a model to understand images, screenshots, design mockups, scanned documents, or even video. There is a key distinction here:
- Native multimodal: The model aligns vision and text during training, so it can accept images/PDFs directly and handle layout understanding, element recognition, and image-text association on its own.
- Assembled pipeline: A separate OCR or format-conversion component first turns the image into text, which is then fed to a text-only model. The chain is longer, errors compound at each stage, and layout structure is often lost during conversion.
For tasks like “design mockup to page code,” “screenshot Q&A,” and “PDF table extraction,” native multimodal can eliminate upstream steps such as OCR and format conversion, significantly reducing engineering complexity and the probability of errors. When evaluating, focus on testing the usability rate of end-to-end results on real business samples, rather than just whether the model “supports image input.”
2.2 Text Reasoning and Instruction Following
Beyond multimodality, you also need to assess core text capabilities: text generation, logical reasoning, structured output (such as reliably producing JSON), and long-instruction following. It is advisable to run a side-by-side comparison using an evaluation set that closely matches your business, covering typical prompts as well as edge cases.
2.3 Context Length and Long-Document Handling
A long context can hold an entire document, codebase, or conversation history in a single pass, but “supports a long context” and “can still retrieve accurately within that long context” are two different things. During selection, specifically test the “needle in a haystack” capability—embed key information deep inside an extremely long input and verify whether the model can reliably locate it.
3. Deployment Form: Public-Cloud API or Private Deployment
3.1 Public-Cloud API: Fast and Elastic
The biggest advantage of calling a cloud-hosted API is no operations overhead, elastic scaling, and pay-as-you-go pricing. You can scale up or down on demand when traffic spikes, without buying your own GPUs. Platforms like Baidu AI Cloud typically provide complementary capabilities such as model services, container platforms, logging, and monitoring/alerting, making it easy to quickly assemble a complete application pipeline.
3.2 Private Deployment: Data Sovereignty and Compliance
When the workload involves sensitive internal documents, customer privacy, or strict compliance requirements, data cannot leave the internal network. In this case, prioritize private deployment: the model runs in your own data center or dedicated environment, and data never crosses the boundary throughout. Private deployment means you must take on GPU procurement/leasing, inference framework selection, quantization and compression, monitoring, and operations—with higher up-front and ongoing operational costs.
3.3 The Role of Open Models in Private Deployment
Open (open-weight) models are a key prerequisite for private deployment—only by obtaining the weights can you truly deploy the model in your own environment. When selecting, confirm: whether the target model offers open weights, whether the license permits commercial use, and whether a version exists in a size that fits your hardware. Open weights let an organization both deploy privately and quickly validate through a cloud-hosted API when needed, enabling a flexible switch between “validation” and “production.”
4. Performance Evaluation: Throughput, Latency, and Concurrency
4.1 Understanding Throughput (tokens/s)
Decode throughput determines “how fast words come out.” 100 tokens/s is roughly 70–80 Chinese characters per second (the exact conversion depends on the tokenizer; for Chinese, one token typically covers less than one character). Many flagship models decode at around 30–40 tokens/s and slow down further on complex long text, which has a clear impact on the interactive experience. The “typewriter effect” that users perceive is essentially governed by this number.
4.2 Time to First Token and Generation Latency
The experience is determined by two kinds of latency together:
- Time to first token (TTFT): from sending the request to emitting the first token, affected by queuing and prefill;
- Generation latency: the interval between each subsequent token, determined by throughput.
For real-time interactive scenarios, you must constrain both at the same time.
4.3 Concurrency and Rate Limiting
High concurrency is a hard requirement for consumer-facing businesses. You can use Little’s Law to roughly estimate resources: required concurrency ≈ request arrival rate (QPS) × per-request processing time.
# Example: estimating the theoretical minimum concurrency based on Little's Lawdef estimate_concurrency(qps: float, latency_seconds: float) -> float:"""qps: peak request arrival rate (requests/second)latency_seconds: average end-to-end time per request (seconds)return: minimum number of concurrent connections needed to sustain this load"""return qps * latency_seconds# Example: peak 20 QPS, average 4 seconds per requestneeded = estimate_concurrency(20, 4)print(f"Theoretical minimum concurrency is about: {needed:.0f}") # about 80
Similarly, generation time can be estimated directly by dividing the number of output tokens by throughput:
# Example: estimating single-turn generation time from throughputdef estimate_generation_time(output_tokens: int, throughput_tps: float) -> float:return output_tokens / throughput_tps# A response generates 600 tokens, measured decode speed is 90 tokens/sprint(f"Estimated generation time: {estimate_generation_time(600, 90):.1f} seconds")
In engineering, you also need to pay attention to server-side rate limiting and queuing strategies to prevent traffic spikes from overwhelming the backend.
5. Cost Modeling: Don’t Look at Unit Price Alone
Per-token price is only part of the cost. A complete model includes at least:
- Inference cost: input/output token price × call volume;
- Resource cost: GPU depreciation, electricity, and data center under private deployment;
- Engineering cost: building upstream multimodal preprocessing, retries, caching, monitoring, and alerting;
- Personnel and operations: model iteration, quality regression, and incident response.
A solution with higher throughput that returns calls faster often needs fewer instances to support the same QPS, gaining an advantage in total cost of ownership (TCO). Selection should account for the total bill, not compare unit price to unit price.
6. Scenario-Based Selection Strategy
6.1 High-Concurrency Consumer-Facing Workloads
The priority is usually: throughput > concurrency capacity > unit price. Users wait staring at the screen, so speed is experience. Such scenarios are better suited to solutions that reliably run at high throughput, support high concurrency, and keep unit price under control, combined with elastic scaling to handle traffic fluctuations.
6.2 Batch Automation and AI Agents
When you want to build AI agents at low cost and large scale, running automation tasks across multiple accounts and scenarios, the evaluation focuses on:
- Cost per task and overall throughput;
- Stability and retryability (batch tasks allow rerunning failed ones, but this must be controllable);
- Reliability of structured output (agents need stable tool calling/JSON);
- Long-context capacity (when you need to carry a large amount of context).
These offline or near-real-time tasks are insensitive to time to first token but rely heavily on cost and concurrent throughput—making them the home turf of “high throughput + low cost” solutions.
7. Go-Live Decision Checklist
When selecting, check off each item:
- Requirements: Is multimodality mandatory? Does it involve sensitive data?
- Capability: Have you run end-to-end evaluation on real samples? Does long-context retrieval meet the bar?
- Deployment: Is a public-cloud API fast and economical enough, or is private deployment required? Does the target model offer open weights, and does its license permit commercial use?
- Performance: What are the measured throughput, time to first token, and concurrency behavior at your target QPS?
- Cost: Are you comparing unit price or TCO, including resources and operations?
- Fallback: Do you have monitoring/alerting, rate limiting, degradation, and rollback plans?
In one sentence: first lock in the capability floor and deployment form using business constraints, then make the final call with quantifiable metrics—throughput, latency, concurrency, and total cost. Translate marketing claims into engineering metrics, and selection stops relying on gut feeling and starts relying on data.
