Low Utilization in Large-Scale AI Compute Clusters: Causes, Metrics, and Efficiency Improvements

From Buying GPUs to Using GPUs: Why Large Compute Clusters Can Sit Idle

According to public reports, an AI company founded in 2023 built Colossus 1 in Memphis, Tennessee, in 122 days, creating one of the world’s largest single-site compute clusters. Its valuation reportedly rose from zero to USD 250 billion within two years, and it received a USD 200 million U.S. Department of Defense contract. However, industry disclosures indicate that the average utilization of Colossus 1 was only about 11%, far below the industry average of around 40%. At the same time, its core model had limited presence in the developer ecosystem, and its main revenue depended on subscriptions and value-added services on social platforms.

In February 2026, it was merged into a rocket and space company at a combined valuation of USD 1.25 trillion. After its training workloads moved to the newer Colossus 2, all compute capacity of Colossus 1 was reportedly leased in full by a competitor AI company. That company’s pre-IPO valuation reached USD 1.2 trillion and at one point exceeded OpenAI as the world’s most valuable AI company.

Underlying this commercial story is an engineering question: the value of compute assets ultimately does not depend on how many GPUs were purchased, but on the product of GPU count, utilization, and task efficiency. If any one of these three factors approaches zero, overall output collapses. This article examines the causes of low utilization in large-scale AI compute clusters from three perspectives: technical metrics, scheduling mechanisms, and cost accounting. It also provides practical paths for improvement.

1. Three Kinds of “Utilization” Should Not Be Mixed: Metric Definitions Determine Optimization Direction

When discussing “utilization,” at least three non-equivalent definitions exist in industry practice.

Metric	Meaning	Data Source	Main Use
Occupancy	The proportion of GPUs allocated to tasks over time	Scheduling system allocation records	Evaluate whether resources are idle
Activity	The proportion of time GPU compute units are busy	Device management and metrics collection components	Identify bottlenecks inside tasks
Effective Compute Utilization, or MFU	The ratio of actual floating-point operations completed per second to the theoretical peak of the hardware	Training framework logs plus hardware peak specifications	Measure training efficiency

The key point is that the product of these three metrics represents real effective compute. A cluster may show occupancy of 100%, activity of 60%, and MFU of only 20%, which means effective compute is only about 12%. If one only looks at “all cards are occupied,” capacity will be seriously overestimated.

Therefore, the first step in diagnosing utilization is not parameter tuning, but unifying metric definitions. For training tasks, MFU should be the primary metric. For interactive development and debugging tasks, occupancy and activity should be primary. For online inference, cost per unit of effective output should be primary. When different workloads run on the same cluster, statistics should first be separated by queue or tenant, and then aggregated with weights.

2. Five Structural Causes of Low Utilization

Low utilization in large-scale clusters is rarely caused by a single factor. More commonly, it results from errors stacking up across construction, scheduling, communication, data, and engineering processes.

1. Mismatch Between Supply Structure and Workload Structure

Hyperscale training clusters are usually driven by investment cycles. Construction takes a long time, procurement volumes are large, and the actual workload at launch may consist mainly of inference services, small-model fine-tuning, algorithm validation, and development and debugging. These workloads typically require fewer GPUs per task and run for shorter periods. If scheduling still assumes training-only, whole-card allocation, and exclusive use, the result is a situation where many cards are occupied but few are actually computing.

2. Coarse Scheduling Granularity and Severe Resource Fragmentation

Whole-card exclusive use plus whole-node allocation is the most common default strategy. A debugging task that only needs two GPUs may still occupy an entire node. For distributed jobs requiring multiple tasks to start together, the job must wait until all required nodes are ready before it can run, while already allocated nodes spin idly. After a task ends, scattered idle cards may not be enough to form the scale required for a full training job, creating fragmented idle capacity.

3. Mismatch Between Parallelism Strategy and Network Topology

Once a model exceeds the memory of a single GPU, one must introduce tensor parallelism, pipeline parallelism, data parallelism, sequence parallelism, or even expert parallelism. The finer the split, the more frequent communication becomes. If a parallel group spans racks or even availability zones, communication traffic must pass through high-speed networks whose bandwidth and latency are significantly worse than intra-node interconnect. GPUs then frequently wait for communication to complete, causing MFU to drop sharply. Intra-node interconnect bandwidth is usually much higher than inter-node network bandwidth, and this is a hard constraint that must be considered first when designing parallelism.

4. Data Supply and Checkpoint Writing Become Bottlenecks

Training requires continuously feeding data into GPUs. Common failure points include: data decoding and preprocessing taking too long on the CPU, insufficient storage read bandwidth, request amplification caused by small files and random access, checkpoint writes competing with training reads for bandwidth, and checkpoints being so large that write time exceeds the mean time between failures. If any of these links cannot keep up, GPUs will periodically wait for data.

5. Engineering Losses and Missing Feedback Loops

Repeated experiments consuming resources, jobs remaining unreclaimed for a long time after failure, developers debugging directly on the cluster, and long-tail jobs staying alive indefinitely all erode effective time. A deeper cause is the lack of a feedback loop. If costs are not broken down by team, project, or task type, waste cannot be quantified, and no one is accountable for utilization.

3. Scheduling and Resource Pooling: From Whole-Card Exclusive Use to On-Demand Sharing

The first lever for improving utilization is at the scheduling layer. The principle is to align task granularity with resource granularity: small tasks use slices, large tasks use topology-aware placement, and idle resources can be borrowed.

1. Queues, Quotas, and Priorities

Queues should be divided by business attribute, and each queue should have a guaranteed quota, an upper limit, and a priority. High-priority tasks can preempt idle resources from low-priority tasks. Preempted tasks need checkpoint support so they can resume from the interruption point. A borrowing mechanism can allow idle quota to be temporarily lent to other queues and reclaimed when the original queue needs it, avoiding the situation where quota is held but not used.

2. Gang Scheduling and Topology Awareness

Distributed training jobs should use gang scheduling, meaning all required resources are satisfied at once before the job starts, avoiding idle spinning caused by partial readiness. Combined with topology awareness, nodes for the same job should preferably be allocated within the same rack or network layer to reduce cross-layer communication overhead.

3. GPU Slicing and Sharing

• Compute time-slice sharing: Suitable for development, debugging, and low-concurrency online inference. When multiple processes share the same GPU, interference may occur, so isolation is needed when latency stability is critical.
• Hardware-level partitioning: Some data-center GPUs can be split into multiple isolated instances. This is suitable for inference and small-scale tasks and can provide deterministic resource boundaries.
• Inference replica sharing: Inference services can batch multiple requests into the same computation batch, significantly improving single-GPU throughput, provided memory is sufficient for the required cache space.

4. Tidal Reuse

Training and inference workloads often have staggered peaks. During the day, a baseline capacity can be reserved for online inference, and surplus capacity can be opened to training jobs. At night, the allocation can be reversed. The key is prediction and elastic scaling, not static partitioning.

The following configuration snippet shows an abstract representation of this approach at the scheduling-system level. Field semantics are illustrative and should be adjusted to the actual scheduler.

# Queue: guaranteed quota + upper limit + borrowing/preemption
queues:
  - name: train-priority
    priority: high
    guaranteed: 256        # Guaranteed number of GPUs
    max: 384               # Upper limit after borrowing idle resources
    allow_preempt: true
  - name: train-normal
    priority: normal
    guaranteed: 384
    max: 640
    preemptible: true      # Can yield idle resources to high-priority queues
  - name: inference-slo
    priority: critical
    guaranteed: 192
    max: 256
    preemptible: false     # Online services do not participate in preemption
  - name: dev-debug
    priority: low
    guaranteed: 32
    max: 64
    share_mode: gpu_slice  # Small tasks use sliced instances
placement_policy:
  gang_scheduling: true    # Gang scheduling: start only when all resources are available
  topology_aware: true     # Prefer nodes in the same rack
  bin_pack: true           # Compact packing to reduce fragmentation

4. Improving Training Efficiency: Increase Effective Compute per GPU

After occupancy improves, the next step is to ensure that occupied time actually produces results. The core metric on the training side is MFU. Its numerator is the actual floating-point operations completed by the model in each iteration. Its denominator is the product of GPU count, single-GPU peak compute, and elapsed time.

Improvement directions include:

• Choose a reasonable parallelism combination: Prefer data parallelism when it can solve the problem. If splitting is required, prefer intra-node tensor parallelism and cross-node pipeline parallelism. Frameworks with strong communication-computation overlap capability can appropriately loosen split granularity.
• Overlap communication with computation: Bucket gradients, trigger communication for a portion of gradients as soon as they are computed, and continue backward computation. When conditions allow, use hierarchical aggregation to reduce the number of cross-layer transfers.
• Balance mixed precision and recomputation: Low-precision computation can significantly improve throughput, but high-precision master weights must be maintained to ensure stable convergence. Recomputation trades extra compute for memory savings. It is clearly beneficial when memory is tight and should be disabled when memory is abundant to avoid reducing MFU.
• Optimize the data pipeline: Use columnar or sharded packed data formats to reduce small files and random access. Parallelize prefetching and decoding, and make data reading an asynchronous pipeline with computation.
• Refine checkpoint strategy: Set checkpoint frequency according to the principle that write time should be significantly shorter than the mean time between failures. Use asynchronous saving or sharded saving to reduce interference with training.

The following snippet shows a general diagnostic approach for low utilization: correlate scheduling allocation records with device activity metrics, identify tasks with long occupancy but low activity, and use them as input for an optimization list.

def find_idle_holders(allocation_records, utilization_samples):
    """Find tasks that hold resources for a long time but show low activity as optimization inputs"""
    idle = []
    for task in allocation_records:
        samples = utilization_samples[task.job_id]
        # Proportion of time GPU compute units are busy during the entire execution period
        busy_ratio = sum(
            s.busy_ms for s in samples if s.util > 50
        ) / max(task.duration_ms, 1)
        if task.duration_hours >= 2 and busy_ratio < 0.3:
            idle.append({
                "job_id": task.job_id,
                "team": task.owner,
                "gpu_hours": task.gpu_hours,
                "busy_ratio": round(busy_ratio, 3),
                # Output a recommended action to close the discovery-location-remediation loop
                "suggestion": "slice" if task.gpu_count <= 4 else "review_parallelism",
            })
    return sorted(idle, key=lambda x: x["gpu_hours"], reverse=True)

5. Improving Inference Efficiency: Turning Idle Compute into Service Capacity

When a cluster is large but its core model lacks competitiveness, one common option is to turn the capacity into external supply. A rocket and space company leasing an idle cluster to a competitor AI company is essentially using external demand to fill an internal load gap. For this path to work, the cluster must have stable external service capability.

Inference-side optimization methods include:

• Continuous batching: Instead of waiting for a full batch of requests to arrive, let a finished request exit immediately and allow new requests to join at any time. This significantly improves GPU time utilization.
• Paged cache management: Split intermediate caches for each request into fixed-size blocks allocated on demand. This reduces reserved waste and allows far more concurrent requests per GPU.
• Prefix cache reuse: For requests sharing the same prompt prefix, cache and reuse already computed intermediate results to reduce repeated computation.
• Quantization and distillation: Under acceptable precision constraints, reduce the bit width of weights and activations to lower memory usage and memory access overhead, enabling larger batches.
• Elastic scaling and SLO assurance: Set replica counts based on time to first byte and per-output-byte latency. Scale automatically according to queue length, and set a scale-down cooldown period to avoid oscillation.

6. Multi-Tenant Compute Platforms: Isolation, Metering, and Predictability Are Prerequisites for Sharing

Offering idle compute externally is not as simple as “hanging cards out.” It requires building a multi-tenant compute platform. Core capabilities include three categories.

• Isolation: Use containers and virtualization to isolate compute, GPU memory, network bandwidth, and storage I/O across multiple dimensions. Combine with network policies to restrict inter-tenant communication and ensure that tasks from different tenants do not interfere with each other.
• Metering: Charge by GPU-hour, GPU-memory-hour, network egress traffic, or effective output volume. Retain auditable usage details to provide a unified basis for internal cost allocation and external settlement.
• Predictability: Publish clear SLOs and quotas externally. Use priority queues to prevent a single tenant from occupying all resources. Also retain capacity buffers to handle sudden peak demand.

It is important to emphasize that external supply can improve asset returns, but it cannot replace model capability. When internal models lack competitiveness in the developer ecosystem and among enterprise customers, GPU rental can only delay losses; it cannot reverse competitive position.

7. Efficiency Governance: Making Utilization a Sustainable Operational Metric

Utilization is not a one-time project but an operational capability that requires long-term management. It is recommended to establish a three-layer metric system:

• Asset layer: Overall cluster occupancy, fragmentation rate, and the share of idle GPU-hours.
• Task layer: MFU, average wait time, number of preemptions, failed rerun ratio, and checkpoint time share.
• Cost layer: Cost per unit of effective training throughput, cost per unit of effective output, and the share of idle cost.

Supporting mechanisms include: publishing a weekly utilization dashboard to teams; automatically alerting on and reclaiming continuously low-activity tasks; allocating idle cost to the corresponding project budgets; and introducing utilization forecasts into procurement decisions to avoid planning capacity based solely on peak load.

8. Phased Implementation Roadmap

Phase	Timeframe	Key Actions	Quantifiable Goals
Phase 1: Measurement	0–1 month	Unify metrics; connect device, scheduling, and task-level indicators	Establish a cluster-level utilization baseline
Phase 2: Scheduling Governance	1–3 months	Queues and quotas, gang scheduling, topology awareness, idle reclamation	Noticeable reduction in idle GPU-hours and converging queue wait time
Phase 3: Task Efficiency	3–6 months	Parallelism tuning, communication overlap, data pipeline optimization, checkpoint optimization	Improved MFU for typical training tasks
Phase 4: Serviceization	6–12 months	Inference serviceization, elastic scaling, multi-tenant isolation and metering	Stable external supply capacity

It should also be noted that utilization metrics should serve business goals, not replace them. Over-pursuing high occupancy may increase queue time for critical experiments and hurt online service SLOs. A more reasonable approach is to set a composite target of utilization, SLO, and cost per unit of effective compute.

Conclusion

The story of Colossus 1 illustrates a simple principle: compute is capital-intensive and technology-intensive infrastructure, and its return depends not on construction speed but on operational efficiency. An average utilization rate of 11% means that nearly 90% of GPU-hours did not produce effective computation. Meanwhile, even the industry average of 40% shows that mature teams still have significant room for improvement.

For technical teams, the practical path is to first unify measurement definitions, then optimize scheduling, tasks, and cost feedback loops in sequence. For business decision-makers, replacing “how many GPUs to buy” with “how much to spend per unit of effective compute” is a more reliable basis for judgment.