From Compute to Applications: A Practical Guide to End-to-End LLM Deployment
This article explains how teams can turn LLM capabilities into stable, cost-effective, and secure applications. It covers compute adaptation, inference and token optimization, model-as-a-service (MaaS), AI coding, industry deployment patterns, and practical engineering recommendations.
1. Background: The moat is shifting from models to products and engineering
Over the past two years, the industry has largely focused on model parameter counts and benchmark scores. However, when it comes to real-world deployment, the core question has changed: how can organizations deliver large language models reliably, cost-effectively, and securely into business workflows? In other words, a leading model is only an entry ticket. Real competition takes place across the entire engineering chain of compute supply, inference efficiency, service stability, and scenario adaptation.
A mature deployment system typically consists of three layers:
- Compute layer: provides heterogeneous acceleration resources and completes hardware-software co-adaptation.
- Model layer: handles training/fine-tuning, inference acceleration, and model service packaging.
- Application layer: delivers ready-to-use product capabilities for scenarios such as finance, government, and office productivity.
Only when these three layers work together can “compute, models, and applications” form a closed loop instead of operating in isolation.
2. Compute layer: engineering challenges of diverse accelerator adaptation
The diversification of accelerator hardware—especially the growing variety of domestic accelerators—is a clear trend, but it also brings significant adaptation costs. Engineering teams should focus on four major issues:
- Operator and compilation adaptation: Different accelerator cards vary significantly in operator support, instruction sets, and compiler toolchains. Teams need to perform operator replacement, graph optimization, and compilation backend adaptation at the inference framework layer to avoid the situation where a model “runs, but does not run fast”.
- Precision and numerical consistency: Mixed precision (such as FP16/BF16) and quantization can significantly affect throughput, but different hardware may behave differently at the numerical level. Alignment regression must be performed using a unified precision validation set.
- Memory and bandwidth constraints: When single-card memory is limited, large models need to be partitioned across multiple cards using tensor parallelism, pipeline parallelism, or layered offloading.
- Hardware-software decoupling: A unified abstraction layer can hide underlying hardware differences, allowing upper-layer model services to achieve “adapt once, deploy across multiple environments”.
Engineering recommendation: Add precision alignment tests and performance benchmark tests to CI. Run the same set of benchmarks after any hardware or framework upgrade to avoid adaptation regression risk.
3. Inference and tokens: improving throughput while controlling cost
The cost of large language model inference is essentially token cost. To optimize both latency (time to first token) and throughput (output tokens per second), common techniques include:
- Tokenization basics: Tokenizers (such as BPE/SentencePiece) determine token granularity, directly affecting context length and billing units. This should be clarified at the requirements stage.
- KV Cache management: Caching historical Key/Value pairs avoids repeated computation. Combined with paged memory management, this can significantly improve concurrency.
- Continuous batching: Dynamically batching different requests improves GPU utilization and is key to increasing throughput.
- Quantization and speculative decoding: Weight quantization reduces memory usage, while speculative decoding uses a smaller draft model to accelerate generation from a larger model.
A rough cost estimation example (for capacity planning, not precise metering):
# Estimate the token size of a single request for capacity planning and rate limiting (illustrative)def estimate_tokens(text: str, chars_per_token: float = 1.5) -> int:return int(len(text) / chars_per_token)prompt_tokens = estimate_tokens(user_input)print(f"Estimated usage: approximately {prompt_tokens} tokens")
4. Model serving: building stable MaaS capabilities
When models need to be exposed as services, a layer of Model-as-a-Service (MaaS) capability is required. This usually includes:
- Unified gateway: authentication, rate limiting, quotas, and routing, hiding differences across multiple models.
- Elastic scaling: automatic scaling based on metrics such as QPS, memory usage, and latency, balancing cost and user experience.
- Multi-model management: a single platform manages models with different parameter sizes and from different vendors, making canary releases and switching easier.
- Observability: logs, metrics, and tracing together provide real-time monitoring of latency and error rates.
Example deployment configuration:
# Illustrative inference service deployment configuration (hardware-independent)service:name: llm-inferencereplicas: 3resources:accelerator: gpu # Can be replaced with accelerator cards from different vendorsmemory: 32Giruntime:dtype: float16 # Mixed precisionmax_batch_size: 16max_sequence_length: 8192autoscaling:min_replicas: 2max_replicas: 20target_qps: 50
On the client side, a unified general interface can decouple business code from the underlying model:
import requestsdef chat(endpoint: str, api_key: str, messages: list, model: str) -> str:headers = {"Authorization": f"Bearer {api_key}"}payload = {"model": model,"messages": messages,"temperature": 0.7,"max_tokens": 1024,}resp = requests.post(f"{endpoint}/chat/completions",headers=headers, json=payload, timeout=30)resp.raise_for_status()return resp.json()["choices"][0]["message"]["content"]
5. Application layer: AI coding and integrated industry solutions
AI coding is one of the fastest-growing application scenarios. Its core challenge is not the model itself, but context engineering: how to accurately provide the model with repository structure, dependency relationships, and coding standards so that it can generate suggestions that are useful and adoptable. Engineering teams need to handle context trimming, multi-file retrieval, and result validation to avoid generating code that “looks reasonable but cannot actually run”.
In compliance-sensitive industries such as finance and government, key points for integrated solutions include:
- Private or dedicated deployment: data remains within a controlled environment, with compute and models operating in a closed loop.
- Security and access control: fine-grained data access controls and audit trails.
- Industry adaptation: on top of general capabilities, domain fine-tuning and knowledge augmentation improve the accuracy of professional Q&A and document processing.
6. Deployment recommendations and best practices
- Scenario before technology: Start with high-frequency, fault-tolerant scenarios such as document processing and coding assistants, then gradually move into core business workflows.
- Decouple compute and models: Use a unified abstraction layer to hide hardware differences and avoid being locked into a single compute vendor.
- Make evaluation routine: Build a joint benchmark for precision, latency, and cost so that every iteration can be regression-tested.
- Cost-first design: Consider token costs and elastic scaling strategies during architecture design.
Overall, deploying large language models is a coordinated engineering effort across “compute—model—application”. Whoever makes this chain more stable, more cost-effective, and easier to use will gain an advantage in the next stage of product competition. Baidu AI Cloud’s engineering practices in compute adaptation, model serving, and industry applications also support this point: integrated engineering capability is increasingly becoming a key variable that determines deployment success.
