How to Build the Compute and Data Foundation for Production-Ready Large Models
Moving a large language model into production requires more than buying compute. This article examines the four foundational assets—compute, data, engineering, and organization—and provides practical formulas for estimating training and inference costs, comparing rented and self-built infrastructure, and accounting for utilization, caching, batching, and failed experiments.
1. Beyond the Capital Rush, the Foundation Sets the Ceiling
AI is a capital-intensive industry. During training, organizations must purchase or rent compute, maintain clusters, and pay for electricity. The data lifecycle requires continuous cleaning, labeling, and evaluation. During inference, operating expenses continue to grow with user demand. These investments are long-term and volatile, and several rounds of early-stage funding may not be enough to carry a business to stable profitability.
Recently, some model and compute companies have moved toward public listings. Going public can provide more consistent access to financing, while disclosure requirements may also help attract talent and build customer trust. For hardware companies such as chip vendors, architectural upgrades take place over multi-year cycles, making stable funding expectations particularly important. Progress by individual companies does not necessarily represent an industry-wide trend; any assessment should be based on public disclosures.
Capital can buy compute, but it cannot directly buy scheduling efficiency, a data governance system, or experience optimizing inference. What teams need to accumulate are four layers of assets:
| Layer | Core output | Acceptance metrics |
|---|---|---|
| Compute | A unified pool of schedulable heterogeneous resources | Resource utilization, queue time, failure recovery time |
| Data | Continuously updated, high-quality corpora and business data | Cleaning pass rate, data freshness, evaluation-set coverage |
| Engineering | Standardized workflows for training, inference, and evaluation | Experiment reproducibility, release time, rollback success rate |
| Organization | Collaboration mechanisms across algorithm, platform engineering, and business teams | Requirement delivery cycle, time to close cross-team issues |
The sections below cover training, inference, scheduling, and deployment. Each section includes formulas that can be applied directly.
2. Training Costs: First Distinguish Renting from Building
Training costs are large, often one-time expenditures. The most common estimation error is mixing up the number of GPUs with GPU-hours. The formulas below use total GPU-hours as the billing unit, with each quantity counted only once.
# Step 1: Total GPU-hours for one successful training run# The sum of the runtime of all GPUscard_hours = F_total / (P_peak * MFU) / 3600# Step 2: Total GPU-hours after accounting for failed and aborted experimentsK = card_hours * (1 + N_fail * f)# Renting: the unit price already includes electricity, so do not add power costs separatelyC_rent = K * p_gpu# Self-built infrastructure: GPU depreciation + electricity based on wall-clock time + operationswall_hours = K / GC_self = K * c_dep + wall_hours * P_kw * p_elec + C_ops
Variable definitions
| Variable | Meaning | Unit |
|---|---|---|
F_total |
Total training computation | FLOPs |
P_peak |
Peak compute of one GPU | FLOPs/second |
MFU |
Model FLOPs utilization, calculated from measured throughput; ranges from 0 to 1 | Dimensionless |
card_hours |
Total GPU-hours for one successful training run | GPU-hours |
N_fail |
Number of failed or aborted experiments | Runs |
f |
Average fraction of a complete training run consumed by each failed experiment; ranges from 0 to 1 | Dimensionless |
K |
Total GPU-hours after accounting for failed experiments | GPU-hours |
G |
Number of accelerator cards participating in training | Cards |
wall_hours |
Wall-clock training time, assuming all cards are fully loaded throughout | Hours |
p_gpu |
Rental price per GPU-hour, including electricity | Currency/GPU-hour |
c_dep |
Depreciation cost per GPU-hour for self-built infrastructure | Currency/GPU-hour |
P_kw |
Total cluster power, including servers and cooling | Kilowatts |
p_elec |
Electricity price | Currency/kWh |
C_ops |
Fixed costs for facilities, networking, personnel, and other operations | Currency |
MFU is defined as the actual computation completed during a time window divided by the peak computing capacity of all GPUs over the same window. The denominator must include time; otherwise, the ratio is meaningless. In practice, select a stable operating window and multiply the number of tokens processed per second by the computation required per token to estimate actual compute. For dense Transformers, a commonly used approximation is:
F_total ≈ 6 × parameter count × training token count
GPU-hours should not be entered based on intuition. They should be derived from total computation and measured MFU. For rented infrastructure, the unit price already includes electricity, so the power cost must not be counted again.
The following example is hypothetical. All figures are provided only to demonstrate the formulas and do not represent the actual prices or performance of any vendor. Assume total computation of 1e24 FLOPs, a peak performance of 3×10^14 FLOPs per second per GPU, an MFU of 40%, and 1,000 GPUs. Failed experiments are not included for now (N_fail = 0).
- GPU-seconds:
1e24 / (3e14 × 0.4) ≈ 8.33×10^9seconds - GPU-hours:
8.33×10^9 / 3600 ≈ 2.31×10^6GPU-hours - Wall-clock duration:
2.31×10^6 / 1000 ≈ 2,315hours, or approximately 96.5 days - Rental scenario: at 20 currency units per GPU-hour,
C_rent ≈ 2.31×10^6 × 20 ≈ 46.3 millioncurrency units - Self-built scenario: assume each GPU costs 250,000 currency units, is depreciated over five years, and has an average utilization rate of 80%. Then
c_dep = 250000 / (5 × 8760 × 0.8) ≈ 7.1currency units per GPU-hour. The depreciation component is approximately 16.4 million currency units. At a total cluster power of 1,000 kW and an electricity price of 0.6 currency units per kWh, electricity costs approximately 1.39 million currency units. The two components total approximately 17.8 million currency units, excludingC_ops.
These results should not be used to draw a direct conclusion about which approach is better. Rental pricing already includes facilities and electricity, while self-built depreciation depends on the utilization assumption. A proper infrastructure decision must include operations and personnel costs and replace the assumed utilization rate with the organization’s actual utilization data.
3. Inference Costs: They Scale with Usage and Are Easy to Underestimate
Inference is an ongoing expense. Within a stable throughput range, doubling the number of daily requests will roughly double the daily cost. For self-built inference infrastructure, daily cost can be estimated as follows:
# Daily inference cost estimate# Unit: currency units per daydaily_cost = R * (I * (1 - h) * p_in + O * p_out) / 1e6 / b * (1 + r)
Variable definitions
| Variable | Meaning | Range or unit |
|---|---|---|
R |
Daily request volume | Requests/day |
I |
Average input tokens per request | Tokens |
O |
Average output tokens per request | Tokens |
h |
Share of input tokens that can hit the prefix cache | 0 to 1 |
p_in |
Baseline input price before optimization | Currency/million tokens |
p_out |
Baseline output price before optimization | Currency/million tokens |
b |
Throughput multiplier from continuous batching | At least 1; must be measured under the target workload |
r |
Redundancy rate, including retries and duplicate generations | At least 0; for example, 0.05 represents 5% |
Three points require attention when using this formula.
First, h is the proportion of the input covered by a common prefix multiplied by the cache hit rate. Prefix caching can reuse only the shared portion of the input, such as a system prompt, not the user’s question.
