Local Deployment of Trillion-Parameter-Scale LLMs: A Practical Guide to Load Testing and Capability Evaluation

    This article walks through a hands-on case study of deploying a full-precision trillion-parameter-scale LLM (about 1.6T parameters) on local infrastructure, where a round of load testing and capability evaluation produced disappointing overall results. Drawing on that real deployment, it breaks down a reusable workflow covering load-testing methodology, metric interpretation, bottleneck localization, and evaluation design. Readers will learn how to separate service performance (TTFT, TPOT, prefill/decode and total throughput, error rate) from model capability (logic, instruction following, tool calling, structured output), why both dimensions must be tested together, how deployment topology and GPU memory constraints shape the numbers, and how a tiered text-to-SQL evaluation set combined with dual-track scoring reveals a model's true ability. The headline finding—36% under strict scoring versus 88% under tolerance—shows the gap came almost entirely from formatting and precision rather than logic.

    1. Introduction: Why Run a “Two-Dimensional Health Check”

    As open-source weights for hundred-billion- and trillion-parameter models keep landing, more teams are trying to deploy the “complete, full-weight” version into their own compute environments. The full version skips aggressive weight quantization and avoids various accelerated distillation schemes, so in theory it best represents a model’s true capability. But in one real deployment, a tester ran two full rounds—one with a speed-testing tool and one with an answer-quality tool—and the overall verdict was “unsatisfactory”: some results were even worse than the lighter fast variants in the same model family.

    That conclusion looks simple, but unpacking it reveals two independent problems:

    • The service doesn’t run smoothly: as concurrency rises, first-token latency degrades sharply and throughput drops instead of climbing.
    • The answers aren’t well-formed: the business semantics are correct, yet the model loses points en masse on formatting and precision requirements.

    Looking at just one dimension easily leads to a partial or even wrong conclusion. Focus only on throughput and you might think “the speed is fine”; focus only on average accuracy and you’ll miss the cliff-edge drop across difficulty levels. Using this real test as a sample, this article breaks load-testing methodology, data interpretation, bottleneck localization, and evaluation design into a reusable workflow—helping developers quickly pinpoint issues in similar local deployments and establish an acceptance baseline.

    2. Two Distinct Problems: Service Performance vs. Model Capability

    2.1 Service Performance: Can You Use It, and How Well?

    Inference service performance measures how fast and at what scale the system can complete requests under a given hardware and concurrency load. Typical metrics of interest include:

    • Time to first token (TTFT);
    • Average time per output token (TPOT);
    • Prefill throughput;
    • Decode throughput;
    • Total system throughput;
    • Request error rate.

    This dimension is largely determined by the deployment setup—hardware, parallel sharding, serving-framework scheduling parameters, memory headroom, and so on—and is unrelated to how “smart” the model itself is. Put differently, the same model on a different scheduling configuration can perform drastically differently.

    2.2 Model Capability: Are the Answers Correct and Reliable?

    Model capability measures the task-level correctness of outputs, including logical reasoning, instruction following, tool calling, and structured output. It is influenced by many factors: the model weights themselves, prompt design, context length, and calling method (whether multi-turn tool calling is allowed), among others. The same model with a different prompt set can also see large swings in score.

    2.3 Why Both Must Be Tested Together

    In real business scenarios, the two constrain each other:

    • However strong the model capability, if latency under concurrency is unacceptable, it’s hard to put into an online pipeline;
    • However good the service performance, if pass rates on key tasks are too low, users still won’t buy in.

    So local-deployment acceptance should include at least two rounds: one to stress service performance, one to evaluate model capability, then place both sets of data side by side in the same acceptance table.

    3. Deployment Topology Defines the Test Boundary: Start with the Memory Ledger

    3.1 The Memory Ledger for Trillion-Parameter-Scale Models

    To discuss performance, you first have to acknowledge physical constraints. Assume a model with roughly 1.6 trillion (1.6T) parameters loaded at BF16 precision; the weights alone require:

    1. 1.6e12 parameters × 2 bytes/parameter ≈ 3.2 TB of GPU memory

    And that doesn’t include:

    • KV cache: grows linearly with context length and concurrency;
    • Activations and temporary buffers: peak during the prefill stage;
    • Framework overhead: memory fragmentation, communication buffers, and so on.

    As a result, this class of model generally can’t be launched at full precision directly on a single 8-GPU server. It requires multi-node, multi-GPU deployment combined with tensor parallelism (TP), expert parallelism (EP, applicable to mixture-of-experts architectures), or pipeline parallelism (PP). The higher the degree of parallelism, the larger the cross-GPU and cross-node communication overhead—and that overhead directly eats into a portion of the theoretical compute.

    For a tester, this means: load-testing results are always “results under a specific deployment topology.” Under different parallel strategies and different inter-GPU interconnect bandwidths, the numbers can differ significantly. So the report must state the deployment topology clearly; otherwise the data is neither reproducible nor comparable across setups.

    3.2 Memory Consumption Breakdown

    Component Influencing Factors Variation with Concurrency/Length
    Model weights Parameters × precision Fixed
    KV cache Layers × hidden dim × context length × concurrency Grows linearly
    Activations/temporary buffers Tokens per batch, sequence length Fluctuates with batch size
    Framework overhead Fragmentation, communication buffers Roughly fixed

    4. Load Testing Methodology: Metrics, Variable Control, and a Sample Script

    4.1 Define the Metrics First

    Before load testing, definitions must be unified—otherwise numbers from different people can’t be compared:

    • TTFT (Time To First Token): the time from the client issuing a request to receiving the first output token, covering queuing, network transmission, and prefill;
    • TPOT (Time Per Output Token): after the first token, the average interval per output token;
    • Prefill throughput: input tokens processed per unit time; governs how fast the model “eats” the prompt;
    • Decode throughput: output tokens generated per unit time; governs how fast it “speaks”;
    • Total system throughput: the sum of effective output-token rates across all requests under concurrent load;
    • Goodput: throughput counting only requests that meet business constraints (e.g., TTFT < threshold)—closer to real user experience than “raw throughput.”

    4.2 Control the Variables: Change One at a Time

    • Fix input-length tiers, e.g., 2k / 4k / 8k / 16k / 32k;
    • Fix output length (e.g., max_tokens=512) and temperature (temperature=0);
    • Fix the concurrency gradient (1 / 2 / 4 / 6 / 8);
    • Warm up with a few requests at each tier before official timing;
    • Repeat each combination over multiple rounds, take the median, and note the dispersion;
    • Isolate external interference—for example, shut down other tasks and pin the network path.

    4.3 Sample Load Testing Script

    1. import time
    2. import requests
    3. URL = "http://<inference_service_address>/v1/chat/completions" # OpenAI-compatible generation endpoint, illustrative
    4. def stream_once(prompt: str, max_tokens: int = 512):
    5. payload = {
    6. "model": "my-local-llm", # placeholder, replace with the actual deployment name
    7. "messages": [{"role": "user", "content": prompt}],
    8. "max_tokens": max_tokens,
    9. "temperature": 0,
    10. "stream": True, # must enable streaming to measure TTFT/TPOT
    11. }
    12. first_at, last_at, chunk_cnt = None, None, 0
    13. t0 = time.perf_counter()
    14. with requests.post(URL, json=payload, stream=True, timeout=600) as r:
    15. for line in r.iter_lines(decode_unicode=True):
    16. if not line or not line.startswith("data:"):
    17. continue
    18. now = time.perf_counter()
    19. first_at = first_at or now
    20. last_at, chunk_cnt = now, chunk_cnt + 1
    21. ttft = (first_at - t0) if first_at else float("nan")
    22. tpot = (last_at - first_at) / max(chunk_cnt - 1, 1)
    23. return {"ttft": ttft, "tpot": tpot, "chunks": chunk_cnt}

    Note three things: first, you must enable streaming output, otherwise you can’t measure real TTFT and TPOT; second, leave the timeout generous—under long prompts plus high concurrency, TTFT can reach the order of minutes; third, client-side timing only reflects the “end-to-end experience”—server-side queuing and in-batch waiting still need the framework’s built-in metrics to fill in.

    5. Measured Results and Interpretation: Where Does the Speed Dimension Break Down?

    A local-deployment load test of a 1.6T-scale full-parameter model (default serving-framework configuration, GPU memory utilization ~0.91) produced the following:

    Metric Measured Behavior
    Prefill single-concurrency peak 757.7 tok/s (4k input); 629–758 tok/s across 2k–32k, no clear length decay
    Prefill with concurrency 2 concurrent ~ -30% to -45%, 4 concurrent ~ -50%, 6 concurrent ~ -63% (2k: 688 → 228 tok/s)
    Decode single-concurrency 23.8–28.7 tok/s, fairly stable
    Decode with concurrency 8–18 tok/s; drops to an 8–11 tok/s trough at 16k/32k long prompt + 4–6 concurrent; rebounds to 20–28 tok/s at the 32k tier, likely data-point noise
    TTFT Rises linearly with length; at 32k, 51.5s at 1 concurrent → 186.4s at 6 concurrent—queuing is the main cause
    Bottleneck diagnosis Concurrency cap set to 2: above 2, requests queue and long prefills block each other, degrading TTFT and smearing Prefill throughput
    Improvement direction Raising the concurrency cap and batch-token cap can improve high-concurrency throughput, but must be weighed against memory (utilization already 0.91, KV nearly maxed out)

    Key takeaways:

    1. Prefill at single concurrency is decent, but concurrency scaling is poor. Per-request throughput of 629–758 tok/s shows no obvious weakness in hardware compute or interconnect; but at 6 concurrent, per-request throughput falls to only about 228 tok/s—meaning the server isn’t effectively “packing” concurrent requests into the same batch but letting them queue and block each other. This is a classic scheduling-configuration problem, not a hardware one.
    2. TTFT grows linearly with length, as expected. Prefill compute is roughly proportional to input length, so a single-concurrency TTFT of about 51.5s at 32k input is reasonable; but climbing to 186.4s at 6 concurrent, far beyond linear superposition, shows queuing dominates.
    3. The decode side is memory-bandwidth-bound, more so under long context. Each generated token requires reading all weights and the KV cache—a memory-bandwidth-intensive operation. Under long prompts the KV cache footprint is large, and stacking high concurrency on top drops single-request decode speed to 8–11 tok/s, a very poor experience.
    4. The 32k rebound is likely noise. With small samples, long sequences, and high concurrency, timing jitter in a few requests’ scheduling is enough to pull up the average. Such “anomalous rebounds” should be re-tested rather than written straight into conclusions.

    6. Performance Optimization: Scheduling, Memory, and Topology

    6.1 Scheduling Parameters: The Most Direct Step

    • Raise the concurrency cap (max-num-seqs-type parameters): currently set to 2, the direct cause of queuing above 2;
    • Raise the per-batch token budget (max-num-batched-tokens-type parameters): lets multiple short requests merge into the same batch, improving Prefill efficiency;
    • Enable chunked prefill: split long prompts into chunks and mix them with decode requests for scheduling, avoiding one long request blocking the whole batch, and markedly improving the TTFT long tail under high concurrency;
    • Set the max context length sensibly: size it to what the business actually needs instead of maxing it out by default, which frees up a lot of KV memory.

    6.2 Memory: The Ceiling on Concurrency

    Raising the concurrency cap isn’t free. Higher concurrency means a larger KV cache footprint, and utilization of 0.91 is already near the ceiling; pushing further may trigger preemption or swapping, actually lowering throughput. Options include:

    • KV cache quantization (e.g., INT8/FP8 KV), trading a small precision loss for several-fold KV capacity;
    • Prefix caching, reusing the KV of a shared system-prompt prefix to cut redundant prefill;
    • Lowering weight precision (when quality is acceptable), handing the freed memory to the KV cache.

    6.3 Topology and Instance Design

    • If cross-node communication is the bottleneck, consider higher-bandwidth interconnect or a better parallel split;
    • If a single instance can’t satisfy both “low latency” and “high concurrency,” adopt a “multiple replicas + front-end load balancing” design, routing long prompts and short prompts to different instance pools so long requests don’t drag down short ones;
    • Wire load-testing metrics into a logging service and monitoring/alerting to sustain continuous observation of “latency percentiles, throughput, KV usage”—not a one-off test that gets discarded.

    6.4 One Practical Takeaway

    Optimization must “change one variable at a time.” Raising the concurrency cap and the batch-token cap together makes improvements impossible to attribute. Recommend re-running the same load-testing script after each change, recording three items—TTFT median/P95, decode throughput, and error rate—to build a comparable optimization ledger.

    7. Capability Evaluation Design: Tool Calling + Text-to-SQL as the Yardstick

    7.1 Why Choose This Kind of Task

    Text-to-SQL is an ideal yardstick for a large model’s overall capability:

    • It requires understanding business semantics and table structure;
    • It requires multi-step reasoning (multi-table JOINs, window functions, cumulative share);
    • Outputs can be automatically machine-verified, so judging is objective and reproducible;
    • It naturally fits a “generate SQL → execute → verify” tool-calling loop.

    7.2 Tiered Evaluation Set

    The sample set has 25 questions, split by difficulty into trivial / easy / medium / hard, covering single-table queries, multi-table joins, window functions (LAG, ROW_NUMBER), cumulative share, and quarter-over-quarter (QoQ) comparisons. The benefit of tiering is that a total score masks problems—only tiering reveals where the “capability breakpoints” occur.

    7.3 Judgment Criteria Must Be Strict and Public

    The sample judging rule is: a question PASSES only if row count, column count, column names, and first-row values all match exactly. This is the “official” standard; the upside is that it’s objective and reproducible, the downside is that it’s extremely strict on format—write logic that’s correct but omit one ROUND(…, 2) and it still scores 0. So the report must present both “strict judging” and “relaxed numeric tolerance (e.g., 1e-6)” results, or the model’s true level is easy to misread.

    7.4 Measured Results

    Dimension Conclusion
    Total score 9/25 PASS (36.0%): trivial 8/8, easy 1/2, medium 0/2, hard 0/13
    SQL generation All 25 questions completed the tool-calling loop autonomously (generate SQL → execute on a local embedded analytical database → verify → results_ok); complex multi-table JOIN + window functions (LAG/ROW_NUMBER/cumulative share) were all written correctly, with only 2 structurally incorrect
    Main point loss ROUND precision: often omits the ROUND(…, 2) the prompt requires; under exact matching, 15 of 20 questions were “logic right, one point off on format”
    Structural failures Multi-table + discount CTE question (final returned all_rows column name mismatched), QoQ window-function question (0 rows + column name mismatched), channel-definition question (expected a 73.3% Reseller-channel share, model computed Internet)
    Token consumption trivial/easy average 2 steps (~2.2k input + 0.2k output tok); hard average 6.1 steps, one question maxing at 17 steps / 55.5k tok, another 14 steps / 55k tok
    Evaluation note Judged strictly by the official standard; relaxing precision to 1e-6 raises the pass rate to about 22/25 (88%)
    Optimization suggestions Strengthen the system prompt to enforce strict adherence to the prompt’s ROUND/format requirements; provide sample SQL for window-function questions

    The most valuable thing about this data is: 36% and 88% describe the same model. The gap comes almost entirely from “format/precision” rather than “logic.” For engineering, this means:

    • The model isn’t “unable to do it”—it’s “unwilling to deliver as required”;
    • The fix path can be light—tighten the prompt, add an output post-processing layer (auto-fill ROUND, unify column names and order), and separate “logic score” from “format score” in the evaluation script;
    • Reporting only 36% badly underestimates the model’s capability; reporting only 88% hides the delivery risk.

    7.5 Failure Mode Classification and Countermeasures

    Failure Type Typical Symptom Impact Countermeasure
    Precision/format Omits ROUND(…, 2), column name mismatch Logic right but judged as fail Strict prompt constraints + output post-processing
    Structure Wrong returned column names/count, 0 rows Direct failure Define the output contract clearly, provide column-name samples
    Semantic definition Wrong channel definition Wrong result Inject a definition dictionary into the prompt, few-shot examples
    Step/token blowup hard questions 14–17 steps, 55k tok High cost, easy to time out Cap max steps, cache intermediate results

    8. General Recommendations for Production

    1. Two-dimensional acceptance table: put “performance metrics + capability metrics” in the same table; failing any item means not passing.
    2. Fix the service first, then evaluate the model: if scheduling parameters aren’t fixed, capability evaluation is simultaneously distorted by timeouts, truncation, and tool-calling failures, so the scores can’t be trusted.
    3. Dual-track evaluation standard: use strict judging as the go-live gate and tolerance judging to diagnose capability boundaries—report both.
    4. Manage prompts like code: format-related point losses are often solvable with prompts and post-processing, at a cost far lower than switching models.
    5. Observe continuously, don’t load-test once: wire in monitoring, alerting, and logging; watch P95/P99 latency, KV usage, and error rate to catch the “fine normally, collapses at peak” hazard.
    6. Leverage general cloud capabilities: when running similar validation on general-purpose platforms such as Baidu AI Cloud, you can use a general container platform to orchestrate multi-GPU tasks and elastic scaling, use object storage to host model files and test datasets, and use logging and monitoring services to accumulate load-testing and evaluation results—turning a “one-off test” into a sustainable regression process.

    9. Conclusion

    The complete conclusion from this local deployment: on the service side, concurrent scheduling is the hard flaw; on the model side, format adherence is the soft flaw. The former is solved through parameter tuning and memory optimization; the latter through prompt constraints, output post-processing, and evaluation-standard design. For developers, the most valuable takeaway isn’t “how many tok/s a given model hit,” but mastering a reusable method: first clarify the deployment topology and memory boundary, then use unified metrics to press out the bottleneck, and finally use a tiered task set plus dual-track judging to assess true capability. Fix this method into your workflow, and for any model on any hardware you can quickly produce a trustworthy acceptance conclusion.