Performance Testing Boundaries

Before analyzing the 500+ TPS metrics, it’s crucial to define the testing constraints:

• Model Configuration: 8×B300 architecture (8 parallel B300 inference units)
• Concurrency Control: Single-user concurrency=1 (no multi-user contention)
• Caching Strategy: 92% prefix cache hit rate (92% of inputs reuse KV cache)
• Task Type: Pure decoding interaction (excluding pre-filling, tool calls, or code execution)

The test workload comprised 13-round interactions with ~220 tokens per initial output and 973 incremental tokens in subsequent rounds. Through KV cache reuse, only 85.8K out of 1.04M input tokens were new (91.75% reuse rate). While this design highlights decoding performance, it differs from production environments that involve tool calls and long-context processing.

System-Wide Optimization Stack

Achieving 500+ TPS required end-to-end optimizations from caching to hardware layers:

1. Caching Layer: Prefix Caching & KV Reuse

Prefix caching stores intermediate computation results (KV cache) to eliminate redundant calculations. At 92% cache hit rate:

• Pre-filling Phase: 80K context directly reuses KV cache
• Decoding Phase: Computes only new tokens’ KV values (90%+ reduction)

# Pseudocode: KV Cache Management
class KVCacheManager:
    def __init__(self):
        self.cache = {}  # Stores <prefix_id: kv_cache> pairs
    def get_cache(self, prefix_id):
        return self.cache.get(prefix_id)  # Returns cached data if exists
    def update_cache(self, prefix_id, new_kv):
        self.cache[prefix_id] = new_kv  # Updates cache

2. System Coordination: IndexShare & MTP V2

IndexShare reduces redundant index operations between DSA (Decoding State Accelerator) and MTP (Multi-Token Prediction) modules by 30%+ on 8×B300 architecture.

MTP V2 enables processing 5 tokens per validation step (vs. traditional token-by-token validation), boosting throughput by 400% through:

• Parallel candidate token processing
• Dynamic programming for optimal path selection
• Spec V2 scheduling to hide preparation operations

3. Operator Optimization: Matrix Operations & Memory Access

Long-context processing challenges:

1. KV Matrix Selection: Rapidly locating relevant rows in million-scale matrices
2. Small Matrix Multiplication: Decoding typically involves <64×64 matrices

Solutions:

• TopK-V2 Algorithm: Reduces matrix selection time from O(n) to O(log n) via sparse indexing
• Kernel Fusion: Combines multiple CUDA kernels into single operations
• BF16 GEMM Optimization: Maintains precision while improving throughput

Hardware acceleration strategies:

• NVFP4 Quantization: Compresses model weights from FP16 to 4-bit (75% memory reduction)
• FP8 KV Cache: Balances precision and bandwidth requirements
• TEP=8 Configuration: Processes 8 decoding requests in batch for maximum GPU utilization

Performance Optimization Chain

The 500+ TPS achievement results from coordinated optimizations:

1. Caching Layer: 92% prefix cache hit → eliminates 80%+ pre-filling computation
2. Index Layer: IndexShare reduces redundant operations → lowers DSA/MTP overhead
3. Decoding Layer:
   • MTP V2 processes 5 tokens/step → boosts base throughput
   • Spec V2 hides scheduling gaps → eliminates computation cycles
4. Operator Layer:
   • TopK-V2 optimizes matrix selection → shortens long-context processing
   • Kernel Fusion/BF16 GEMM → accelerates small matrix operations
5. Hardware Layer: NVFP4/FP8/TEP=8 → maximizes GPU parallel efficiency

Performance Metric Limitations

The 500+ TPS metric applies under strict constraints:

• Low Concurrency: Single-user concurrency=1 (no multi-user contention)
• Pure Decoding Tasks: Excludes tool calls and code execution
• Fixed Workload: Uses production-like replay testing without dynamic loads

In real production environments, performance may vary. For example:

• At concurrency=16, memory bandwidth may become a bottleneck
• Adding tool calls could increase end-to-end latency by 30%+
  These metrics serve as engineering validation rather than production performance guarantees.

Reusable Optimization Methodology

Developers seeking to improve model inference performance should build optimizations across these dimensions:

1. Caching Strategies:

   • Implement granular KV cache management
   • Design prefix cache eviction algorithms (e.g., LRU-K variants)

2. System Coordination:

   • Identify and eliminate redundant computations between modules
   • Use pipeline techniques to hide scheduling delays

3. Operator Optimization:

   • Develop specialized kernels for small matrix multiplications
   • Implement computation graph-level operator fusion

4. Hardware Adaptation:

   • Select optimal quantization schemes based on GPU architecture
   • Adjust batch sizes to match memory bandwidth

This systematic approach can achieve 200-300 TPS decoding performance even in resource-constrained environments, providing technical guarantees for real-time interactive applications.