TPS Metrics: Definition and Measurement Conditions

While TPS (Transactions Per Second) generally indicates requests processed per second, GLM-5.2’s 500+ TPS claim comes with strict conditions:

• Hardware Configuration: 8×B300 accelerators (8 cards with 300B parameters each)
• Parallelism Settings: TEP=8 (Token Expansion Parameter) with single-user concurrency
• Workload Characteristics: 92% prefix cache hit rate under specific dialogue patterns

Key Measurement Parameters:

• Decode Interactivity: Measures response capability during decoding phase only (excluding TTFT and full task duration)
• Token Increment Calculation: 13-round dialogue with 80K initial context and ~973 new tokens per subsequent round (via prefix compression)
• Quantization Benchmark: Maintains model accuracy using Blackwell architecture’s BF16 mixed-precision quantization

This measurement differs fundamentally from industry benchmarks that include agent workflows like tool invocation and code execution. The 500+ TPS metric specifically reflects pure decoding performance rather than full 80K-context inference capabilities.

Workload Optimization: From Input Compression to Caching

Three mechanisms enable high TPS for long-context workloads:

1. Aggregate Prefix-Cache Compression

Reduces computation by converting 80K inputs into incremental calculations. After initial full-context processing, subsequent requests only compute new tokens (~973 per round), cutting actual computation by 98.7%. This design achieves 92% cache hit rates in dialogue scenarios.

2. Dynamic Token Allocation

The MTP (Multi-Token Prediction) architecture enables batch prediction of 5+ tokens in coding workloads. Compared to single-token generation, this reduces decoding iterations and directly improves throughput.

3. Long-Context Selection Optimization

The improved TopK-V2 algorithm reduces GPU-CPU synchronization overhead by fusing 12 indexer kernels into 4, tripling long-text processing efficiency.

Technical Validation: Industry benchmarks show unoptimized systems achieving <200 TPS under identical hardware, while GLM-5.2 reaches 300+ TPS through workload optimization alone.

Model Architecture Innovations: From Computation Reuse to Parallel Acceleration

GLM-5.2 eliminates serial computation bottlenecks through three core improvements:

1. IndexShare Mechanism

Reduces redundant index calculations by 40% in long sequences through shared index caching across decoder layers.

2. MTP Architecture Upgrade

The dynamic window mechanism adjusts predicted token counts (5-8 tokens per decoding step in code generation) based on input features, improving efficiency by 60% over fixed windows.

3. Mixed-Precision Quantization

BF16 quantization cuts memory usage by 50% while maintaining accuracy. Combined with CuTe DSL kernel optimization, this delivers 2.3× speedup for small-batch GEMM operations.

Performance Progression:

• Baseline model: 180 TPS (8×B300)
• After IndexShare: 260 TPS
• After MTP improvements: 380 TPS
• Final quantization optimization: >500 TPS

System-Level Synergy: From Runtime to Kernel Optimization

Four technologies enable hardware-software co-design breakthroughs:

1. Spec V2 Scheduler

Eliminates CPU-GPU synchronization gaps through asynchronous task pipelines, achieving >95% resource utilization and reducing end-to-end latency by 35%.

2. Prologue Fusion Technology

Fuses 12 long-context kernels into 4 composite kernels, reducing memory access and PCIe traffic. This improves GPU utilization from 68% to 89% in 80K-context scenarios.

3. BF16 GEMM Acceleration

CuTe DSL enables automatic kernel tuning for small-batch matrix operations, delivering 2.3× speedup over generic libraries in 32×32 matrix scenarios.

4. Dynamic Batching Strategy

Maintains GPU utilization >85% while ensuring P99 latency <500ms through workload-aware batching algorithms, ideal for interactive inference.

Engineering Complexity: Full reproduction requires modifying 200+ model code sections, rewriting 30 core kernels, and adjusting 8 hardware parameters - explaining why third-party hardware struggles to replicate 500+ TPS.

Technical Insights and Industry Applications

GLM-5.2 reveals key principles for AI engineering:

1. Full-Stack Collaboration: Isolated optimizations yield <20% gains; system-level tuning enables order-of-magnitude improvements
2. Workload-Driven Design: Dialogue and code generation require fundamentally different caching and parallelism strategies
3. Transparent Reporting: Detailed disclosure of hardware configurations and optimization scripts provides more value than closed benchmarks

Enterprise Recommendations:

• Implement prefix caching for long-context scenarios
• Configure dynamic MTP window sizes based on workload patterns
• Adopt Spec V2-style schedulers for resource efficiency
• Perform kernel-level tuning for target hardware platforms

GLM-5.2 demonstrates that significant inference performance gains remain achievable through workload reconstruction, architectural innovation, and system-level co-design. This methodology provides a reproducible path for AI engineering implementation, with value extending beyond individual performance metrics.