Eight-Card LLM Inference Stress Testing and Tuning Guide for Chinese AI Accelerator Cards

    This article provides a practical guide for stress testing and tuning large language model (LLM) inference services deployed on a single node with eight Chinese AI accelerator cards. It covers applicable scope, baseline and benchmarking methodology, bottleneck localization, and parallelism, communication, and compilation tuning. The guidance targets dense models from 7B to 70B parameters and interactive workloads such as online Q&A, intelligent customer service, and enterprise knowledge bases. It explains how to record baselines, design test sets, interpret utilization, memory, communication, TTFT, TPOT, and queueing metrics, and select parallelism strategies. All values are examples or starting references and must be recalibrated for the actual card type, driver, engine, and model before production.

    1. Applicable Scope

    This article applies to LLM inference services deployed on a single node with eight Chinese AI accelerator cards. The recommended model range covers dense models with 7B to 70B parameters. Business scenarios are mainly interactive requests such as online Q&A, intelligent customer service, and enterprise knowledge bases. Engine, driver, and compiler versions depend on the actual environment; this article does not specify vendor models.

    Models with long contexts over 32K tokens or mixture-of-experts architectures require separate testing. The thresholds and parallelism plans in this article cannot be directly applied to them.

    All numerical values in this article are examples or starting reference values and do not represent measured results for any card type. Before going live, recalibrate them for your own card type, driver, engine, and model.

    2. Baseline and Benchmarking Methodology

    Without a baseline, you cannot determine whether optimization is effective or explain which change produced the gain. Fill in the table below before each test. The values in the table only show the required format.

    Record Item Example Value Filling Requirement
    Model and version A 7B-class conversational model, weight version v1.2 Specify version number, weight source, and checksum
    Inference precision FP16 Note quantization scheme separately, including weight and KV cache precision
    Input length distribution Short text 40%, medium 40%, long 20% Based on real request logs from the last seven days
    Output length Average 256 tokens, maximum 1024 Fix max_tokens and record the actual generation length distribution
    Concurrency curve 8, 16, 32, 64, 128 10-minute steady-state window per level; requests arrive according to a Poisson distribution
    Card count and topology 8 cards, single node, measured inter-card bandwidth Use vendor bandwidth tools to measure point-to-point and all-to-all bandwidth in GB/s
    Engine and runtime versions Engine, driver, compiler versions Record each item; any change is treated as a new baseline
    Parallel configuration 4-card tensor parallel × 2 replicas Correspond one-to-one with Section 4
    SLO thresholds See Section 5 Baseline and optimized plans must use the same thresholds
    Warm-up and repeats 5 minutes warm-up after service start, 3 formal rounds Record each round separately; use the median for the conclusion

    The test set is divided into four groups by input length, with a minimum sample size for each group.

    • Short-context set: input no more than 512 tokens, at least 2,000 samples. Used to observe scheduling overhead and first-token response.
    • Medium-context set: input from 512 tokens to 4K tokens, at least 2,000 samples. Represents typical online business.
    • Long-context set: input from 4K tokens to 32K tokens, at least 500 samples. Used to analyze memory usage and KV cache hit behavior.
    • Mixed-distribution set: input lengths sampled according to production log proportions, at least 5,000 samples. Closest to real load and the main basis for acceptance.

    Sampling method: sample requests uniformly by time from the last seven days of production logs, first remove records containing personal information or sensitive content, then preserve the original length distribution. Stress test data must not contain personally identifiable information.

    When reporting results, report the average, P50, P95, P99, and peak values together. Reporting only the average hides tail latency problems.

    3. Bottleneck Localization

    In eight-card deployments, performance is usually determined jointly by compute efficiency, memory, and inter-card communication. The table below lists monitoring metrics, collection methods, and abnormal threshold references for each problem type. The thresholds are example starting values and must be calibrated by card type, engine, and model.

    Problem Type Monitoring Metric Collection Method Abnormal Threshold Reference (Example)
    Compute efficiency Compute unit utilization (%) Vendor monitoring tool sampled at 1 s; use the 60 s steady-state average Average below 50% and latency exceeds target: check batching and scheduling
    Memory Memory usage, KV cache usage (%) Sample every second; record peak and average Peak above 90% or allocation failures: lower the concurrency limit or maximum sequence length
    Communication Collective communication time as a share of a single decoding step (%) Engine profiler; sample one complete decoding step Share above 25%: check parallelism strategy and topology
    Time to first token TTFT P99 (ms) Stress test client records per request Exceeds 80% of the business upper limit: check prefill batch size
    Decoding speed TPOT (ms/token) Same as above More than 10% worse than baseline: check compiled artifacts and operators
    Queueing Server-side queueing time P99 (ms) Server request logs Above 30% of end-to-end P99: concurrency exceeds processing capacity

    Troubleshoot in the following order. At each step, examine only one class of metrics and proceed only after the conclusion is clear.

    1. Is the success rate below 99.5%? If yes, check the error type. For memory allocation failures, lower concurrency or shorten the maximum sequence length. For timeouts, check queueing and timeout parameters. For abnormal process exits, check driver and runtime logs. If no, proceed to step 2.
    2. Is the P99 end-to-end latency over target? If yes, check queueing time. If queueing is high, add replicas or reduce per-replica concurrency. If queueing is low, proceed to step 3. If no, proceed to step 5.
    3. Is compute utilization low? If yes, check whether batching is too small, scheduling overhead is too high, or input lengths are too dispersed. If no, proceed to step 4.
    4. Is the communication share high? If yes, check whether tensor parallelism crosses PCIe or NUMA nodes, and consider reducing the parallelism degree or using same-card replicas. If no, check whether compiled artifacts and operators fall back to generic implementations.
    5. Has effective throughput reached the target? If yes, proceed to acceptance in Section 5. If no, rerun the concurrency curve to find the inflection point.

    4. Parallelism, Communication, and Compilation Tuning

    Parallelism Strategy Selection

    The parallelism method depends on model size, context length, inter-card bandwidth, and number of replicas. The following three combinations are common in single-node eight-card scenarios.

    • If a single card can hold the model and high concurrency is needed: prefer data parallel replicas, for example, 8 cards as 8 replicas, or 4 cards per group as 2 replicas. Communication overhead is minimal.
    • If a single card cannot hold the model: use tensor parallelism. Choose the smallest degree among 2, 4, and 8 that can accommodate model weights and KV cache, and keep the same tensor parallel group within the same high-speed interconnect domain.
    • Mixture-of-experts models: require expert parallelism, and all-to-all communication time must be tested separately. This article does not provide general parameters.

    Pipeline parallelism is mainly used in cross-node scenarios. Single-node eight-card deployments usually do not need it.

    Fallback: If communication accounts for more than 25% in the 4-card tensor parallel configuration, change the tensor parallel configuration.