How Cloud-Based Always-On Agents Are Reshaping Compute Architecture

1. The Four Stages of AI Agent Evolution

Many people wonder which application will become the primary user entry point, but for infrastructure teams, a more pressing question is: how should machines be provisioned when AI agents are always online? To answer this, we can first look at the evolution of AI application forms.

2023: Chatbot Q&A. Large language models focused on Q&A. Users input prompts, the model generated text or code, and the execution of actions was still completed by humans. API calls were typically stateless, with resources released immediately after response generation. The workload was characterized by short-lived, bursty connections.

2024 to 2025: Task-Oriented Agents. Agents began to break down multi-step tasks, call external APIs, read context, and perform multi-step reasoning. However, interactions were still single-trigger, and session states were cleared after the task ended. To support multi-step tasks, platforms started keeping containers alive for the duration of the task.

Early 2026: Localized AI Assistants. An open-source framework drew significant developer attention, and several tech companies subsequently launched AI assistants running on user computers. These could read local files, recognize screenshots, and deeply integrate into user workflows. The client side required compute power, while the cloud side needed high-frequency model inference APIs, forming the prototype of edge-cloud synergy.

Late 2026 Onwards: Cloud-Exclusive VMs. Leading AI companies began piloting isolated cloud runtime environments for agents, freeing them from user computers. This stage is still in early validation, and specific progress depends on company announcements. For users, the product transformed from a passive tool to a proactive assistant; for infrastructure, the workload pattern changed significantly.

2. New Workloads Introduced by Always-On Agents

Consider a hypothetical scenario: before the owner of a coffee chain wakes up, a cloud-based agent has already pulled the latest data from the point-of-sale system, e-commerce platform, customer service email, payroll system, and bank account. It generates a morning business briefing and highlights the change in unsubscribe rates after a shipping fee adjustment. This is a constructed example for illustrative purposes, not a tested case.

Such applications commonly use a “proactive polling” mechanism. Even without user commands, the system wakes up in the background every few hours to check emails, calendars, and business systems. This imposes three new requirements on data centers:

1. Long-Term Keep-Alive: Agents need to maintain persistent processes rather than destroyable functions. Underlying containers or VMs must guarantee 24/7 high availability.
2. High-Frequency Wake-Ups: Timer-triggered and event-triggered actions will become the norm. Infrastructure requires fast cold start speeds and efficient resource hot-loading capabilities.
3. Environment Isolation: Each user has an exclusive runtime environment. Networks and data must be isolated, and storage must be encrypted.

3. How Compute, Storage, and Network Must Adjust

Compute: Logical Persistence, Physical Elasticity

Allocating a dedicated GPU for each user is usually cost-prohibitive. Relying entirely on a serverless architecture makes it difficult to maintain session context and local environment states. A more feasible approach is to enhance GPU slicing and pooling capabilities: allocate more compute when the user is active, and reduce allocations or share instances with others during polling intervals.

The key is distinguishing between “logical persistence” and “physical persistence.” Once the agent’s state is persisted, the compute instance can scale down and be restored before the next polling cycle. However, if the agent needs to receive messages in real-time, the instance cannot scale completely to zero. In this case, an event gateway is needed to receive requests in the background and wake up the corresponding instance.

Storage: Three-Tier Division

• Object Storage: Stores unstructured data such as historical reports and attachments. It uses lifecycle policies to transition long-inactive data to low-frequency storage.
• Block Storage or Distributed File Systems: Mounts the agent’s working directory, ensuring that data pulled during polling can be quickly written to disk and processed.
• Vector Database: Used for retrieving historical memory. It must maintain data consistency with block storage to avoid retrieving deleted or expired memories.

Network: Outbound Traffic as the Main Pressure Point

During polling, an agent will simultaneously request dozens of external systems, generating massive outbound traffic and concurrent connections. Three things require attention: whether the concurrent connection limit of the NAT gateway is sufficient, whether outbound requests per tenant are rate-limited, and whether cross-region APIs can reduce latency through dedicated lines or edge caching.

4. Trade-offs Among Three Deployment Models

Model	Use Case	State Persistence	Cold Start	Primary Cost Risk
Always-on VM	Requires a full desktop or long-running tasks	Good	Slow	Still billed when idle; low resource utilization
Container + Persistent Volume	Most always-on agents requiring on-demand scaling	Good, relies on volumes and snapshot mechanisms	Relatively fast	Requires well-designed scale-down strategies and volume recycling
Serverless Function	Stateless data fetching, Webhook processing	Poor, requires external storage for sessions	Fast	At high frequencies, per-invocation costs may be higher

In most scenarios, Container + Persistent Volume is the most balanced choice. Serverless functions are better suited for short tasks like external data fetching, while always-on VMs are ideal for scenarios that genuinely require a full desktop environment.

5. Container Deployment Example

The following example illustrates the deployment structure for a single user. Each user has a persistent volume to save their working directory. The Deployment uses the Recreate strategy to prevent two replicas from simultaneously mounting a ReadWriteOnce volume. The CronJob is solely responsible for scheduled polling triggers and does not host the persistent session.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: agent-workspace-user-001
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: agent-user-001
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector: