Beyond Compute: How Large Models Can Quickly Reach the Leading Tier

    This article examines why some large-model teams can approach leading performance despite limited compute. It covers training-cost accounting, sparse architectures such as Mixture of Experts, scaling-law-driven planning, post-training, data engineering, systems optimization, inference cost reduction, hardware adaptation, open-source ecosystems, organizational practices, model selection criteria, and the remaining gaps behind the leading tier.

    1. The Threshold Is Losing Value

    Over the past year, a notable phenomenon has emerged: within a short period, multiple teams have released new models whose public benchmark results and real-world experience are close to those of international leaders. Three common traits stand out:

    • Limited starting resources: Many teams are not traditional compute-heavy players, and their training scale is far smaller than that of leading international organizations.
    • Diverse architecture choices: Dense models, sparse expert models, mixture-of-experts designs, and multimodal fusion are being pursued in parallel, and all have produced models that entered the leading tier.
    • Fast iteration: Generations are often only months apart, and the slope of capability improvement is much steeper than the slope of investment growth.

    This suggests that the yardstick for model capability is shifting from “who has more compute” to “who can convert each unit of compute, data, and engineering investment into more effective capability.”

    2. The Underlying Logic: From Stacking Resources to Competing on Efficiency

    2.1 Training Cost: Two Sets of Books for the Same Thing

    Public estimates of “how much it costs to train a large model” often differ by tens or even hundreds of times. In public discussion, “starting at hundreds of millions of dollars” and “million-dollar scale” coexist. There are three main reasons:

    1. Different scopes: “Million-dollar scale” usually counts only the accelerator-hours consumed by one final training run. “Hundreds of millions of dollars” includes early ablation experiments, failed attempts, data acquisition and cleaning, labor costs, cluster depreciation, and rework from trial and error.
    2. Different hardware assumptions: Converting costs using accelerator purchase prices overestimates the actual expenditure of renting a cluster. Using depreciation of an existing cluster produces a lower result.
    3. Different points in time: Supply and prices for similar accelerators change quickly, so conversion results differ significantly across training periods.

    A practical way to estimate is this: the true total investment in a large model is often several times to more than ten times the cost of the final full training run. What matters is not the absolute number, but whether the cost of acquiring each unit of effective capability continues to fall.

    2.2 Algorithmic Efficiency: Trading Structure for Compute

    ① Sparsely activated architectures. Take Mixture of Experts (MoE) as an example. The model consists of multiple “expert” subnetworks, and each input is routed to only a few experts for computation. This allows parameter scale to expand substantially to carry knowledge, while each computation involves only a small subset of parameters. If total parameters are 100B and about 10B are activated each time, the compute per token is roughly equivalent to a dense model of the same activated scale.

    This brings two key benefits:

    • Compute per token during training drops significantly, allowing teams to explore a larger parameter space with less compute.
    • During inference, the “large parameters, small compute” property makes high-capacity models more feasible for broad deployment.

    The costs are also clear:

    • Memory footprint: All expert parameters must reside in memory, requiring higher memory capacity.
    • Load imbalance: A few popular experts are selected frequently, causing uneven utilization across devices.
    • Communication overhead: With expert parallelism, tokens must be distributed across and gathered from devices.

    As a result, load-balancing loss, capacity factor, and routing strategy all require careful tuning.

    ② Scaling-law-driven training planning. It is widely observed that model loss has an approximately power-law relationship with parameter count, data volume, and compute. Teams can train a series of smaller models at different scales, fit a scaling curve, and then use it to determine the optimal parameter allocation, data mix, and training steps for a large model. This reduces compute wasted by blind trial and error.

    ③ Post-training: using relatively small investment to unlock reasoning capability. Pre-training determines the knowledge base, while post-training determines whether the model can use it. Supervised fine-tuning, preference alignment, and reinforcement learning with verifiable rewards can significantly improve reasoning, coding, and instruction-following ability without large-scale changes to the pre-trained model.

    For tasks such as math and code, where answers can be automatically verified, rule-based verifiers can directly generate reward signals. This lets the model improve its accuracy through large-scale sampling and trial and error. The training cost of such methods is usually far lower than re-pretraining, yet the improvement in complex reasoning capability is quite direct.

    ③ Data engineering. The contribution of high-quality data is often no less important than architectural innovation. Deduplication, quality filtering, source mixing, and curriculum-style data orchestration all directly affect the model’s final capability ceiling.

    2.3 Systems Engineering: Keeping Hardware Busy

    ① Combinations of parallelism strategies. Large-scale training requires partitioning models across multiple devices. Common strategies include:

    • Data parallelism: Each accelerator holds a full model, data batches are split, and gradient synchronization keeps updates consistent.
    • Tensor parallelism: A single layer’s computation matrices are split across accelerators, requiring extra communication in forward and backward passes.
    • Pipeline parallelism: The model is split by layer, and different batches are passed through stages like a pipeline.
    • Expert parallelism: Different experts are placed on different devices, with auxiliary communication for routing.
    • Optimizer state sharding and offloading: Optimizer states are sharded or offloaded to host memory to relieve memory pressure.

    ② Overlapping communication and computation. Whether communication introduced by parallelism can overlap with computation and avoid bandwidth bottlenecks often determines the overall throughput ceiling. Communication topology, the ordering of parallel dimensions, and batch-size choices all need repeated tuning for a specific cluster.

    ③ Low precision and operator optimization. Training with low-precision numerical formats, fusing frequently used operators, and reducing memory reads and writes can all increase effective compute with almost no change to model quality.

    ④ Long-running stability. Clusters with thousands of accelerators inevitably encounter single-point failures during training. Efficient checkpointing, elastic fault tolerance, and rapid fault localization determine the share of “effective training time.”

    A common metric for these efforts is Model FLOPs Utilization (MFU), the ratio of floating-point operations actually used for model computation to the hardware’s theoretical peak compute. Due to matrix operations, communication waits, and memory access, publicly discussed values commonly range from 30% to 50%. Raising MFU from 30% to 45% is equivalent to gaining about 50% more effective compute with the same hardware. With the same hardware and the same model, different levels of systems engineering can lead to nearly a twofold difference in training cost.

    2.4 Inference Engineering: From Capability Available to Cost Acceptable

    Whether a model can enter the leading tier depends not only on how strong it is, but also on whether it can be used at large scale and low cost. Major inference-side optimizations include:

    • KV cache management: Attention computation requires caching key-value pairs for historical tokens, and this overhead grows quickly with length. Efficient block management and reuse can significantly relieve memory pressure.
    • Continuous batching: Dynamically batching different requests instead of waiting for an entire batch to finish can greatly improve throughput.
    • Quantization: Low-bit quantization of weights, activations, or KV cache reduces memory usage and bandwidth pressure.
    • Speculative decoding: A small model first proposes candidate tokens, and a large model verifies them in parallel, reducing latency while preserving the output distribution.
    • Long-context and output-length control: The longer the context, the greater the attention overhead. Trimming context as needed and controlling maximum output length can often reduce cost more effectively than switching models.

    From the perspective of cost per token, hardware purchase cost, actual throughput, cache hit rate, and batching efficiency jointly determine the final price. If any one of these improves significantly, the model has an opportunity to provide service at a more competitive price.

    3. Hardware Adaptation: Domestic Compute Is Closing the Gap

    Compute supply is the most closely watched issue in this round of catching up. With restricted access to high-end accelerators, domestic AI accelerators have gradually become part of the hardware options for training and inference.

    Adaptation work can be roughly divided into four layers:

    Layer Main content Common challenges
    Hardware drivers and runtime Drivers, runtime libraries, device management Version compatibility, stability
    Operators and compilation Basic operator libraries, graph compiler optimization, custom operators Coverage, performance tuning, numerical alignment
    Frameworks and training Distributed communication libraries, parallelism strategies, fault tolerance Large-scale stability, long-running training
    Models and inference Weight conversion, quantization, inference engine adaptation Numerical consistency, throughput and latency

    The core of adaptation is not “whether it can run,” but three things:

    1. Numerical consistency: Model outputs before and after adaptation must be highly consistent; otherwise, numerical alignment issues directly affect quality.
    2. Performance matching: Operator fusion, communication topology, and batching strategies must all be retuned.
    3. Sustainable supply: A unified abstraction layer and automated testing can reduce the repeated adaptation cost of each model upgrade.

    It is important to be objective: hardware adaptation has significantly reduced the risk of “no accelerators available,” but domestic accelerators still have room to improve in ecosystem maturity, operator coverage, and large-scale stability. At the same time, mainstream model architectures are evolving toward greater sparsity, lower precision, and more bandwidth-friendly designs. These structures are inherently more tolerant of hardware interconnect and memory requirements, which objectively lowers the difficulty of cross-hardware migration.

    4. Open-Source Ecosystems: An Amplifier for Catch-Up Speed

    Over the past two years, the source structure of models downloaded by global developers from open-source model repositories has changed noticeably. This trend is an important clue for understanding catch-up speed.

    Open model weights create a positive feedback loop:

    1. Open weights → global developers download, test, and fine-tune
    2. → feedback on defects and improvement suggestions, vertical-domain derivative models
    3. → more real-world validation → the next version is more targeted → open again

    For resource-constrained teams, an open-source base provides three levers:

    • Lowering the entry barrier: Teams do not need to pre-train from scratch and can focus on data, post-training, and scenario adaptation.
    • Sharing engineering results: Inference frameworks, fine-tuning tools, and quantization methods are mostly open source, so engineering teams can quickly build services on mature toolchains.
    • Leveraging global evaluation: Once a model is released, it immediately faces real testing by global developers. Problems surface faster, and iteration is faster.

    At the same time, download volume reflects ecosystem activity more than model capability itself. When evaluating a model, teams should combine task-specific performance, license terms, and actual deployment cost.

    5. Organization and Methods: Turning Efficiency into Repeatable Processes

    Efficiency and iteration speed are essentially a combination of methodology and engineering capability. Practices worth learning from include:

    1. A small but complete training experiment system: First validate hypotheses with small-scale proxy models, then scale up gradually, avoiding large compute consumption for every attempt.
    2. Full-link observability: Logging, monitoring, and alerting across data collection, training, evaluation, and inference so problems can be located quickly.
    3. Integrated engineering and algorithms: Parallelism strategies, kernel optimization, cluster scheduling, and algorithm design advance together rather than in isolation.
    4. Multi-hardware gray-scale validation: Validate multiple hardware options under a unified abstraction layer to avoid strong dependence on a single hardware platform.
    5. Evaluation-driven iteration: Build an evaluation system covering general capability, domain tasks, safety, and cost so that every iteration has a quantifiable basis.

    6. How to Choose the Right Model: A Practical Checklist

    In an environment of rapid model iteration, model selection should focus on the following dimensions:

    Dimension Question to ask Practical advice
    Task fit Does it cover the target scenario? Run small-scale evaluations with real business samples, not just public leaderboards
    Context requirements Input length per request, long-document processing Measure effective information recall under long context
    Deployment form Private, public cloud, or hybrid When sensitive data is involved, prioritize private or dedicated deployment
    Compute conditions Can existing hardware support it? First run small-scale adaptation validation, then decide on expansion
    Licensing and compliance Commercial terms, data sources Confirm license scope and compliance requirements in advance
    Continuous evolution Version iteration frequency, toolchain maturity Pay attention to evaluation systems, monitoring capability, and ecosystem activity

    7. A Sober View: The Leading Tier Does Not Mean There Is No Gap

    The rapid formation of a leading tier is worth recognizing, but several issues still require an objective view:

    • Uncertainty in compute supply: External restrictions remain, and domestic hardware still has room to improve at very large scale.