LLM Inference Optimization on Eight Homegrown AI Accelerators: Up to 53.5% Throughput Gain

    This article presents a case study of LLM inference optimization on an eight-card deployment of homegrown AI accelerators under uniform service-level objectives. Using a quantized sparse model, it compares an eight-way tensor-parallel baseline with an optimized 2-way tensor-parallel, 4-way data-parallel, and expert-parallel configuration, combined with sequence-oriented communication optimization, graph compilation, and kernel optimization. The optimized configuration achieved up to 53.5% higher total throughput under the same service-level objectives. The article also covers baseline limitations, parallel-strategy redesign, communication and compilation tuning, reliable benchmarking, and production deployment recommendations.

    In LLM inference, hardware scale does not automatically translate into service capacity. With the same model and the same number of accelerator cards, throughput, latency, and resource utilization can vary significantly depending on the parallel strategy, communication path, compilation method, and batching parameters.

    For enterprise users, the hard part is rarely a one-time model launch. The real challenge is maintaining stable throughput while meeting service-level objectives such as time to first token (TTFT), end-to-end response time, and concurrency. For LLM deployment on homegrown accelerators, the focus therefore needs to move from “it can run” to “it can be verified, tuned, and reused.”

    1. Test Background: From Basic Deployment to Service-Objective Optimization

    This case study uses a class of sparse large models and optimizes inference on eight new-generation homegrown accelerator cards. The model is quantized to reduce memory footprint and memory-access pressure, and it is deployed using a commonly used distributed inference architecture.

    The test is not focused on a single peak-speed metric. Instead, it compares total throughput between a baseline configuration and an optimized configuration under the same service-level objectives. These objectives may include:

    • TTFT below a specified threshold;
    • per-request end-to-end latency meeting business requirements;
    • stable responses within a certain concurrency range;
    • no obvious queueing or tail-latency jitter caused by throughput optimization.

    Under these constraints, the optimized solution achieved up to 53.5% higher total throughput than the baseline. This result shows that inference performance improvement does not depend only on adding more hardware; proper parallel organization and hardware-software co-design also matter.

    Performance data is closely tied to the specific model version, input and output lengths, quantization method, compilation state, request distribution, and service-level thresholds. The 53.5% figure should therefore be understood as a result under specific test conditions. Actual deployments still need to be revalidated against real workloads.

    2. Limitations of the Baseline: Eight Cards Do Not Mean Eightfold Scaling

    The baseline configuration runs with eight-way tensor parallelism and a single data replica. Tensor parallelism splits parts of the model computation across multiple cards, allowing a single model replica to use larger aggregate memory.

    This approach is relatively straightforward to implement, but it can have several issues in sparse models and high-concurrency scenarios.

    First, the computation path of a single request depends heavily on cross-card communication. After each layer is computed, some intermediate results must be synchronized across devices. If the communication volume is large, compute units may wait for data transfers, reducing accelerator utilization.

    Second, all requests share one model replica. Under concurrent requests, the system mainly relies on dynamic batching to improve compute density, but the number of replicas is limited. Performance is easily affected by single-path request scheduling and differences in sequence length.

    Third, expert routing in sparse models introduces additional data rearrangement. Tokens from different requests may be assigned to different experts. If the parallel method is not designed around the expert distribution, communication can become concentrated, load can become unbalanced, and some devices may sit idle.

    The baseline is therefore suitable as a functional validation and performance baseline, but it is not necessarily the best organization for an eight-card environment.

    3. Core Optimization: Recomposing the Parallel Dimensions

    The optimized solution changes the original eight-way tensor parallelism with a single data replica into a combination of smaller-scale tensor parallelism and multiple data replicas. Specifically, the test uses two-way tensor parallelism, four-way data parallelism, and expert parallelism to run the model.

    The basic idea is to reduce the communication scope of a single request during tensor splitting while increasing the number of model replicas so that multiple requests can be processed in parallel across different replicas.

    3.1 Tensor Parallelism: Controlling Communication Scope per Request

    Tensor parallelism is suitable for splitting model layers with large amounts of computation, but higher parallelism is not always better. As the number of parallel cards increases, computation is split more finely, and the number of cross-device synchronizations and communication overhead may also increase.

    Reducing tensor parallelism from eight-way to two-way shrinks the synchronization scope within each model replica. For compute-intensive layers, this helps reduce communication waits. For the decoding stage, where tokens are generated one step at a time, it can also reduce the share of communication under small-scale computation.

    3.2 Data Parallelism: Increasing Independent Processing Capacity

    Data parallelism replicates the model so that different replicas can process different requests. With two-way tensor parallelism combined with four-way data parallelism, the eight cards are divided into four smaller compute groups.

    Compared with the single-replica mode, multiple replicas can:

    • improve the distribution of concurrent requests;
    • reduce queueing pressure on a single replica;
    • reduce resource waste caused by differences in request lengths;
    • provide more independent scheduling space for dynamic batching.

    However, data parallelism does not take effect simply by copying the model. The scheduler needs to allocate work based on queue length, request priority, input and output lengths, and replica load. Otherwise, some replicas may be busy while others are idle.

    3.3 Expert Parallelism: Matching the Sparse Model Structure

    For sparse models with multiple expert modules, expert parallelism can distribute different experts across different devices or device groups. After routing, tokens are sent to the corresponding experts for computation and then aggregated in the original order.

    The benefit of expert parallelism depends on three factors:

    • whether expert load is balanced;
    • whether token rearrangement and cross-card transfer are efficient;
    • whether communication and expert computation can overlap in a pipeline.

    If experts are unevenly distributed, popular experts may become bottlenecks while some devices remain underutilized. During tuning, therefore, it is not enough to observe average throughput alone. Device utilization, expert hit distribution, and communication wait time should also be monitored.

    4. Communication Optimization: Keeping Transfers from Blocking Computation

    In distributed inference, communication efficiency directly affects the return from parallelism. This is especially true in the decoding stage, where each computation step is relatively small and communication latency is more likely to become visible.

    This case enables a sequence-oriented communication optimization mechanism, allowing computation and data transfer along the sequence dimension to be organized more efficiently. The goal is not simply to reduce all communication, but to use more efficient data splitting, aggregation, and scheduling to reduce unnecessary waiting.

    Common optimization directions include:

    1. Reducing unnecessary data copies;
    2. Combining small communication operations;
    3. Overlapping communication operations with computation stages as much as possible;
    4. Selecting different data layouts based on input length and batch size;
    5. Avoiding repeated creation of temporary buffers in high-frequency decoding loops.

    Communication optimization must be validated against the actual network topology and device interconnect structure. Simply enabling an option in the configuration does not guarantee gains for every workload. For short-input, low-concurrency requests, communication startup overhead may account for a high proportion of time. For long-context, high-concurrency scenarios, communication bandwidth may become the main bottleneck.

    5. Compilation Optimization: Reducing Runtime Overhead

    In addition to parallel strategy, compilation-stage optimization also affects final throughput. Homegrown accelerators typically rely on graph compilation, operator fusion, and kernel optimization to translate higher-level model computation into low-level programs suitable for device execution.

    Key optimization areas usually include:

    • Fixing or converging common input shapes as much as possible;
    • Reusing already compiled computation graphs;
    • Reducing frequent dynamic-shape changes;
    • Fusing frequently used operators;
    • Prioritizing hot paths such as attention, routing, matrix computation, and data rearrangement;
    • Distinguishing between the computation characteristics of the prefill stage and token-by-token decoding.

    The prefill stage usually processes longer inputs and has higher compute parallelism. The decoding stage repeatedly performs small-step computation and is more sensitive to scheduling and communication latency. Optimization should therefore not target only one stage; performance data should be collected separately for both.

    The first run may include compilation and cache-build costs. Tests should include warm-up, and the stable operating phase should be used as the primary basis for comparison. Compilation cache hits should also be recorded so that one-time overhead is not mistaken for inference performance.

    6. How to Build a Reliable Performance Comparison

    To make optimization results meaningful, the baseline and optimized solutions must use consistent test conditions. These should include at least:

    • The same model weights and quantization configuration;
    • The same input-length and output-length distribution;
    • The same concurrency level or request arrival rate;
    • The same service-level objectives;
    • The same warm-up procedure;
    • The same measurement period;
    • The same throughput calculation method.

    The following metrics should also be recorded:

    • Total throughput;
    • TTFT;
    • Average end-to-end latency;
    • P95 and P99 tail latency;
    • Accelerator utilization;
    • Memory usage;
    • Inter-device communication time;
    • Prefill and decode stage time;
    • Queue wait time.

    Throughput without latency constraints can easily lead to conclusions that do not meet production requirements. Average latency without tail latency can also hide instability during peak periods. Performance evaluation should therefore use service-level objectives as boundaries and judge results across multiple metrics.

    7. Production Deployment Recommendations

    In real business environments, tuning can follow a workflow of baseline establishment, bottleneck identification, parallel adjustment, compilation optimization, and regression validation.

    First, use a relatively conservative parallel configuration to launch the model and confirm that weight loading, quantized computation, API calls, and result accuracy are all correct. Next, use monitoring and profiling tools to identify bottlenecks and determine whether the main issue comes from computation, communication, memory, scheduling, or data processing.

    Then adjust tensor parallelism, data parallelism, and expert parallelism one item at a time. Avoid changing too many variables at once, since doing so makes it difficult to determine where gains come from. After parallel adjustments are complete, optimize communication and compilation, and run regression tests across different input lengths, concurrency levels, and request mixes.

    For services that need to run over the long term, the complete deployment configuration should also be saved. This includes hardware count, parallel mode, model version, quantization method, compilation state, batching strategy, and test results. In this way, tuning experience can be turned into reusable deployment templates, reducing the cost of later model migration and environment reconstruction.

    8. Conclusion

    Inference optimization on eight homegrown accelerator cards is essentially a matter of balancing computation, communication, storage, and scheduling. The basic eight-way tensor-parallel configuration can complete deployment quickly, but in sparse models and high-concurrency scenarios it may be limited by cross-card communication, single-replica queueing, and uneven expert load.

    By using a combination of smaller-scale tensor parallelism, multiple data-parallel replicas, and expert parallelism, together with sequence communication optimization, graph compilation, and kernel optimization, the test environment achieved up to 53.5% higher total throughput under the same service-level objectives.

    The key value of this case is not a fixed parameter set, but a transferable methodology: establish a trustworthy baseline, redesign the parallel strategy around the model structure and hardware topology, and then release device capacity through communication, compilation, and scheduling optimization. For teams seeking to deploy LLMs on homegrown compute environments, this is more practical than pursuing hardware scale alone.