Technical Background and Challenges

In the realm of AI large-scale model inference services, performance optimization remains a central concern. As model sizes continue to expand, the demand for computational resources during inference surges, particularly in domestic computing clusters. Efficiently leveraging limited resources to maximize inference performance has become an urgent issue. Traditional inference service stacks often encounter challenges such as low computational efficiency, high communication latency, and insufficient hardware utilization when handling large-scale models, directly constraining overall inference performance.

To address these challenges, a cloud provider has introduced a lightweight flagship multimodal inference service stack based on the GLM-5.3-Flash model. This stack achieves significant performance improvements in domestic computing clusters through a series of technological innovations, with KV broadcast communication optimization technology serving as a pivotal component, offering new insights into inference performance optimization.

KV Broadcast Communication Optimization Technology Explained

KV broadcast communication optimization technology is a core innovation developed by the cloud provider for the GLM-5.3-Flash inference service stack. It aims to reduce communication latency and enhance hardware utilization during inference by optimizing the communication mechanism of KV (Key-Value) caches, thereby improving overall inference performance.

1. Technical Principles

In inference services, KV caches are crucial data structures for storing intermediate model states, and their efficient transmission is vital for inference performance. Traditional communication methods, such as point-to-point or broadcast, often lead to communication bottlenecks in large-scale cluster environments, affecting inference efficiency. KV broadcast communication optimization technology introduces intelligent scheduling algorithms to dynamically adjust the transmission paths and frequencies of KV caches, achieving optimal allocation of communication resources. Additionally, it incorporates technologies like intra-node tensor parallelism, ReplaySSM, W8A8 quantization, and mixed caching quantization with INT8/FP8/BF16 to further enhance computational efficiency and data precision.

2. Architectural Design

The architectural design of KV broadcast communication optimization technology takes into account the characteristics of domestic computing clusters, adopting a layered approach that closely integrates communication optimization with computational optimization. At the bottom layer, intra-node tensor parallelism evenly distributes computational tasks across nodes, maximizing computational resource utilization. The middle layer introduces ReplaySSM technology for efficient management and reuse of KV caches, reducing redundant computations. The top layer employs KV broadcast communication optimization algorithms to dynamically adjust transmission strategies, ensuring efficient use of communication resources.

3. Performance Optimization Strategies

To achieve significant end-to-end performance improvements, KV broadcast communication optimization technology employs multiple strategies:

• Dynamic Scheduling Algorithms: Adjust the transmission paths and frequencies of KV caches based on real-time cluster load, preventing communication bottlenecks.
• Quantization Compression Techniques: Utilize W8A8 quantization and mixed caching quantization with INT8/FP8/BF16 to reduce data transmission volumes and enhance communication efficiency.
• Parallel Computing Optimization: Combine intra-node tensor parallelism for efficient parallel processing of computational tasks, boosting overall computational performance.
• Cache Reuse Mechanisms: Leverage ReplaySSM technology for efficient management and reuse of KV caches, minimizing redundant computations and data transmissions.

Innovations in Cluster Scheduling

Beyond KV broadcast communication optimization technology, the cloud provider has also introduced innovations at the cluster scheduling level, implementing a production-grade Encode–Prefill–Decode (EPD) separation architecture. This architecture decomposes the inference process into three independent stages—encoding, prefilling, and decoding—each of which can be independently scheduled and optimized, enabling efficient parallel processing of inference services.

1. Principles of EPD Separation Architecture

The core idea behind the EPD separation architecture is to decouple different stages of the inference process, allowing each stage to operate and optimize independently. During encoding, the model extracts and encodes features from input data. In the prefilling stage, the model generates intermediate states based on encoding results and stores them in KV caches. Finally, during decoding, the model produces the final output based on KV caches and input data. This decoupling design enables efficient parallel processing of the inference process, enhancing overall inference performance.

2. Architectural Advantages

The EPD separation architecture offers several advantages:

• Efficient Parallel Processing: By decoupling different stages of the inference process, it achieves efficient parallel processing, improving overall inference performance.
• Flexible Scheduling: Each stage can be independently scheduled and optimized, enabling more rational utilization of cluster resources.
• Scalability: The EPD architecture is highly scalable, allowing for easy addition of new computational nodes or optimization of existing node performance.
• Reduced Latency: By optimizing the processing logic and communication mechanisms of each stage, it reduces latency during inference, enhancing user experience.

Performance Improvements and Cost Optimization

By introducing KV broadcast communication optimization technology and the EPD separation architecture, the cloud provider’s GLM-5.3-Flash inference service stack has achieved significant end-to-end performance improvements. Specifically, the stack has demonstrated a 3x increase in end-to-end performance on domestic computing clusters, with hardware efficiency and single-token inference costs nearly matching those of mainstream GPUs. This outcome not only validates the effectiveness of KV broadcast communication optimization technology but also provides new optimization approaches for inference services in domestic computing clusters.

Future Prospects

As AI technology continues to evolve, optimizing inference service performance will remain an ongoing topic. The cloud provider will continue to delve into the field of inference services, exploring new technological optimization pathways. For instance, further research could investigate how to combine KV broadcast communication optimization technology with more advanced quantization compression techniques to achieve higher communication efficiency and lower inference costs. Additionally, exploring the application of the EPD separation architecture in more complex inference scenarios could meet the diverse needs of different users.