How to Build the Compute and Data Foundation for Production-Ready Large Models

    Moving a large language model into production requires more than buying compute. This article examines the four foundational assets—compute, data, engineering, and organization—and provides practical formulas for estimating training and inference costs, comparing rented and self-built infrastructure, and accounting for utilization, caching, batching, and failed experiments.

    1. Beyond the Capital Rush, the Foundation Sets the Ceiling

    AI is a capital-intensive industry. During training, organizations must purchase or rent compute, maintain clusters, and pay for electricity. The data lifecycle requires continuous cleaning, labeling, and evaluation. During inference, operating expenses continue to grow with user demand. These investments are long-term and volatile, and several rounds of early-stage funding may not be enough to carry a business to stable profitability.

    Recently, some model and compute companies have moved toward public listings. Going public can provide more consistent access to financing, while disclosure requirements may also help attract talent and build customer trust. For hardware companies such as chip vendors, architectural upgrades take place over multi-year cycles, making stable funding expectations particularly important. Progress by individual companies does not necessarily represent an industry-wide trend; any assessment should be based on public disclosures.

    Capital can buy compute, but it cannot directly buy scheduling efficiency, a data governance system, or experience optimizing inference. What teams need to accumulate are four layers of assets:

    Layer Core output Acceptance metrics
    Compute A unified pool of schedulable heterogeneous resources Resource utilization, queue time, failure recovery time
    Data Continuously updated, high-quality corpora and business data Cleaning pass rate, data freshness, evaluation-set coverage
    Engineering Standardized workflows for training, inference, and evaluation Experiment reproducibility, release time, rollback success rate
    Organization Collaboration mechanisms across algorithm, platform engineering, and business teams Requirement delivery cycle, time to close cross-team issues

    The sections below cover training, inference, scheduling, and deployment. Each section includes formulas that can be applied directly.

    2. Training Costs: First Distinguish Renting from Building

    Training costs are large, often one-time expenditures. The most common estimation error is mixing up the number of GPUs with GPU-hours. The formulas below use total GPU-hours as the billing unit, with each quantity counted only once.

    1. # Step 1: Total GPU-hours for one successful training run
    2. # The sum of the runtime of all GPUs
    3. card_hours = F_total / (P_peak * MFU) / 3600
    4. # Step 2: Total GPU-hours after accounting for failed and aborted experiments
    5. K = card_hours * (1 + N_fail * f)
    6. # Renting: the unit price already includes electricity, so do not add power costs separately
    7. C_rent = K * p_gpu
    8. # Self-built infrastructure: GPU depreciation + electricity based on wall-clock time + operations
    9. wall_hours = K / G
    10. C_self = K * c_dep + wall_hours * P_kw * p_elec + C_ops

    Variable definitions

    Variable Meaning Unit
    F_total Total training computation FLOPs
    P_peak Peak compute of one GPU FLOPs/second
    MFU Model FLOPs utilization, calculated from measured throughput; ranges from 0 to 1 Dimensionless
    card_hours Total GPU-hours for one successful training run GPU-hours
    N_fail Number of failed or aborted experiments Runs
    f Average fraction of a complete training run consumed by each failed experiment; ranges from 0 to 1 Dimensionless
    K Total GPU-hours after accounting for failed experiments GPU-hours
    G Number of accelerator cards participating in training Cards
    wall_hours Wall-clock training time, assuming all cards are fully loaded throughout Hours
    p_gpu Rental price per GPU-hour, including electricity Currency/GPU-hour
    c_dep Depreciation cost per GPU-hour for self-built infrastructure Currency/GPU-hour
    P_kw Total cluster power, including servers and cooling Kilowatts
    p_elec Electricity price Currency/kWh
    C_ops Fixed costs for facilities, networking, personnel, and other operations Currency

    MFU is defined as the actual computation completed during a time window divided by the peak computing capacity of all GPUs over the same window. The denominator must include time; otherwise, the ratio is meaningless. In practice, select a stable operating window and multiply the number of tokens processed per second by the computation required per token to estimate actual compute. For dense Transformers, a commonly used approximation is:

    1. F_total ≈ 6 × parameter count × training token count

    GPU-hours should not be entered based on intuition. They should be derived from total computation and measured MFU. For rented infrastructure, the unit price already includes electricity, so the power cost must not be counted again.

    The following example is hypothetical. All figures are provided only to demonstrate the formulas and do not represent the actual prices or performance of any vendor. Assume total computation of 1e24 FLOPs, a peak performance of 3×10^14 FLOPs per second per GPU, an MFU of 40%, and 1,000 GPUs. Failed experiments are not included for now (N_fail = 0).

    • GPU-seconds: 1e24 / (3e14 × 0.4) ≈ 8.33×10^9 seconds
    • GPU-hours: 8.33×10^9 / 3600 ≈ 2.31×10^6 GPU-hours
    • Wall-clock duration: 2.31×10^6 / 1000 ≈ 2,315 hours, or approximately 96.5 days
    • Rental scenario: at 20 currency units per GPU-hour, C_rent ≈ 2.31×10^6 × 20 ≈ 46.3 million currency units
    • Self-built scenario: assume each GPU costs 250,000 currency units, is depreciated over five years, and has an average utilization rate of 80%. Then c_dep = 250000 / (5 × 8760 × 0.8) ≈ 7.1 currency units per GPU-hour. The depreciation component is approximately 16.4 million currency units. At a total cluster power of 1,000 kW and an electricity price of 0.6 currency units per kWh, electricity costs approximately 1.39 million currency units. The two components total approximately 17.8 million currency units, excluding C_ops.

    These results should not be used to draw a direct conclusion about which approach is better. Rental pricing already includes facilities and electricity, while self-built depreciation depends on the utilization assumption. A proper infrastructure decision must include operations and personnel costs and replace the assumed utilization rate with the organization’s actual utilization data.

    3. Inference Costs: They Scale with Usage and Are Easy to Underestimate

    Inference is an ongoing expense. Within a stable throughput range, doubling the number of daily requests will roughly double the daily cost. For self-built inference infrastructure, daily cost can be estimated as follows:

    1. # Daily inference cost estimate
    2. # Unit: currency units per day
    3. daily_cost = R * (I * (1 - h) * p_in + O * p_out) / 1e6 / b * (1 + r)

    Variable definitions

    Variable Meaning Range or unit
    R Daily request volume Requests/day
    I Average input tokens per request Tokens
    O Average output tokens per request Tokens
    h Share of input tokens that can hit the prefix cache 0 to 1
    p_in Baseline input price before optimization Currency/million tokens
    p_out Baseline output price before optimization Currency/million tokens
    b Throughput multiplier from continuous batching At least 1; must be measured under the target workload
    r Redundancy rate, including retries and duplicate generations At least 0; for example, 0.05 represents 5%

    Three points require attention when using this formula.

    First, h is the proportion of the input covered by a common prefix multiplied by the cache hit rate. Prefix caching can reuse only the shared portion of the input, such as a system prompt, not the user’s question.