Always-On Agents: Recomputing Cloud Resource Allocation

Moving from a chat box to an always-on cloud work environment is easy to see at the product layer. The change in the underlying resource model is often underestimated. In the past, a user sent a request, the model returned text, and resources rose when the request arrived and fell when the response ended. Always-on agents require a runtime context that can be held for long periods: an isolated compute sandbox, recoverable files and memory, scheduled wake-up and inspection tasks, and the identity and network channels needed to access business systems. For infrastructure, this is not a matter of multiplying chat traffic by some factor. It means rewriting the load from short-lived, bursty inference into a mixed workload that is long-lived, low-duty-cycle, stateful, and egress-heavy.

Form-Factor Evolution and the Load Inflection Point

Placing recent application form factors on one timeline reveals four continuous stages, rather than a single product launch.

The first stage is conversational interaction. The model generates text or code, and execution is still performed by humans. The main infrastructure path is clear: gateway access, inference service scaling, and short-term caching of prompts and responses. Sessions can be very short, and most state can be discarded or retained only in a limited way.

The second stage is task-oriented agents. The system can break down multi-step goals, call tools, and deliver results, but triggering is still mainly based on a single user instruction. When the task ends, processes, temporary directories, and tool sessions are often reclaimed together. The resource curve shows a “spike and fall” pattern, and capacity planning can still rely on request concurrency and token throughput.

The third stage is assistants attached to a user’s existing compute environment. The agent can read and write local or remote workspaces and retain project memory, but its runtime location is tied to the user’s device or an existing development environment. Isolation boundaries and permission inheritance are unstable, and true around-the-clock operation is difficult.

The fourth stage is an independent cloud runtime. Each user or business entity has a recoverable virtual workspace, and the agent does not need to consume terminal compute. The system wakes it in the background according to a schedule or event, checks email, calendars, tickets, and operational data, and writes conclusions back to a briefing or ticket. To the user, a passive tool becomes a proactive assistant. On the resource side, a large number of tenants appear that “look idle but must be quickly recoverable.”

The inflection point is not model parameter scale. It is duty cycle and state residency time. A morning briefing may consume only a few minutes of inference, but to ensure it can run on time, the sandbox image, identity tokens, connection pools, previous inspection watermarks, and intermediate files must remain addressable for hours or even days.

Why the Allocation Model Must Be Rebuilt

Traditional online inference clusters are sized by “concurrent requests × per-request resources.” Always-on agents add at least three different types of load on top of that.

Compute is no longer only inference. Orchestration loops, tool calls, document parsing, browser or remote desktop automation, timers, and queue consumption spend a large amount of time on general-purpose compute rather than accelerators. Inference appears in short bursts: when inspection windows cluster and users are woken up at the same time before work, queuing can occur that is not linearly related to daily active users. If GPU share is still purchased as if for a chatbot, CPU, memory, and local temporary disk will become bottlenecks first. If an accelerator is permanently bound to every idle sandbox, the cost is unacceptable. A more reasonable approach is to separate the control plane, the sandbox data plane, and the inference pool: sandboxes remain lightweight and resident or quickly recoverable, inference is shared elastically through queues, and priority distinguishes interactive requests from background inspections.

Storage changes from an “attachment to logs” into the product itself. Code, spreadsheets, attachments, browser downloads, and long-term memory in a virtual workspace all need to be persistent, snapshot-capable, and isolated per tenant. Object storage is suitable for artifacts and archives, block storage or local high-speed disk is suitable for sandbox runtime, and retrieval-oriented storage is suitable for long-term memory. But not all state should be forced into the same medium. It is also necessary to distinguish rebuildable cache from business conclusions that cannot be lost: inspection watermarks, processed email cursors, and approval records belong to the latter, and losing them can cause duplicate actions or missed checks.

Networking shifts from mostly north-south requests to frequent east-west traffic and egress. The agent needs to read systems such as point-of-sale, online stores, customer support mailboxes, payroll, and accounts. A single inspection is a chain of authenticated API calls and possible web scraping. Egress bandwidth, connection counts, DNS, certificates, and peer rate limits will hit ceilings earlier than model throughput. Private lines or private connectivity, egress proxies, and per-target-domain quotas should enter the capacity model rather than being added as allowlist entries after launch.

A schematic relationship is more useful than treating this as a measured formula:

Daily cost roughly consists of four parts: memory and disk residency for recoverable sandboxes; inspection trigger count multiplied by per-tool-call cost; accelerator resources for inference queuing; and egress traffic plus external API quotas. As the number of idle sandboxes rises, the first item grows linearly. During a concentrated morning wake-up, the latter three items show correlated peaks. Optimizing only one item will not reduce the overall bill structure.

Reference Architecture

An operable always-on agent platform is usually divided into a control plane and a data plane, so that scheduling logic is not written into every sandbox.

The control plane handles task definitions, identity, quotas, wake-up, and auditing. After a user configures a policy such as “summarize operational data at a fixed time every day,” the scheduler writes it into a task table and delivers a message at the scheduled time, rather than running a sleep loop inside the sandbox. This allows idle instances to be frozen, hibernated, or restored from snapshots while scheduling semantics remain intact.

The data plane is the isolated execution environment. Each tenant has an independent filesystem view, process namespace, and network namespace. Sharing the host home directory or reusing another tenant’s browser profile is prohibited. Images should be as small as possible, and mutable state should be placed on mountable volumes to support snapshots and migration.

Inference and tools are shared services accessed through internal queues. The sandbox holds only short-lived credentials to call the tool gateway and does not store master keys for third-party systems over the long term. The tool gateway centrally handles authentication, parameter validation, retries, circuit breaking, and egress auditing.

A schematic wake-up flow can be expressed as pseudocode. It only shows control relationships and does not correspond to any vendor API:

on schedule(tenant, policy):
    if quota.exceeded(tenant):
        emit metric("skip_quota")
        return
    lease = sandbox.restore_or_create(tenant)
    token = identity.issue_short_lived(tenant, policy.scopes)
    job = queue.enqueue(lease, token, policy.checklist)
    # Inference is requested on demand inside the job and returned when done
on job.finish(result):
    state.commit(cursors, artifacts, summary)
    audit.append(who, what, external_calls)
    sandbox.freeze_if_idle(lease)

This flow emphasizes three things: quotas are checked before wake-up, credentials are short-lived, and state commit is separated from instance freezing. Otherwise, “online 24 hours a day” will be implemented as “holding a machine idle 24 hours a day.”

Identity, Permissions, and Security Boundaries

The risk of always-on agents does not mainly come from a single prompt. It comes from the long-term authorization they hold. Being able to access email and business systems while the user sleeps is equivalent to delegating human operating permissions to an unattended process.

Least privilege should be granted by task, not opened all at once by person. A morning briefing only needs to read orders, read cancellation reasons, and write a report. It should not also have write permissions to issue refunds, change prices, or run payroll. Scopes can be declared with the checklist, and the gateway narrows them to specific resources and actions.

Secrets should not enter images or conversation logs. External system credentials should be placed in dedicated secret management and injected at runtime as short-lived tokens. Conversations, tool parameters, and response bodies should be redacted by default. Inspection failures must be replayable, but full bills or personal information should not be written repeatedly into ordinary logs.

Isolation must prevent lateral movement. One tenant’s browser profile, cookies, and download directory must not be scheduled to another tenant. Snapshot encryption, volume-level tenant binding, and blocking sandbox access to cloud platform metadata services are baselines, not optional features.

Human-in-the-loop control remains necessary. Actions involving fund transfers, bulk external sending, or data deletion should stop in a confirmation queue. Background autonomy should cover only read-heavy, write-light, reversible organization and summarization.

Capacity, Observability, and Cost

Capacity planning should be split into four tables: tenant residency, wake-up concurrency, inference bursts, and egress quotas. Then take the tightest constraint rather than using one QPS number to represent everything.

Residency capacity depends on the number of simultaneously recoverable sandboxes, the size of each snapshot, and the restore latency target. If the goal is interactivity within seconds, hot snapshots or a prewarmed pool are needed. If the goal is background tasks at minute-level latency, pulling images and volumes from object storage is acceptable. The unit price differs by an order of magnitude, and the same SLA cannot be applied to all users.

Wake-up concurrency depends on whether schedules cluster. Natural clock times such as the top of the hour and before work create correlated peaks. Jitter, sharding, and fair queues should be added to avoid thousands of inspections simultaneously saturating external mailboxes or payment APIs, and then retrying collectively after being rate-limited by the peer.

Observability should be divided into at least four lines: scheduling (latency, skips, backlog), sandboxes (restore failures, disk, freeze rate), tools (external error codes, retries, quotas), and inference (queue time, batch size, cancellation rate). Looking only at model latency will misdiagnose the bottleneck as accelerators when it is actually egress or state volumes.

Cost governance depends on product policy rather than after-the-fact load testing. It is possible to set per-tenant daily inspection counts, maximum egress bytes, and maximum tool-chain steps; automatically freeze and retain snapshots after an idle threshold; and prioritize interactive sessions over background briefings. While explaining the value of a “proactive assistant,” quotas should make idle operation visible and stoppable.

Rollout Sequence

It is not necessary to build a complete virtual computer in the first step. A more reliable path is to first externalize the state of a single task into persistent volumes and cursor storage and prove that crashes are recoverable. Then introduce a scheduler and freezing to remove idle spinning. Next, consolidate external system access behind a tool gateway and add short-lived credentials and auditing. Finally, split hot pools and cold restores by tenant type and give the morning peak its own inference and egress budget.

Each step should be able to answer three questions with metrics: how many instances are consuming memory while actually having no work; how many external calls one inspection makes on average and whether failed calls amplify retries; and how much of the user-perceived latency comes from restoring the sandbox rather than from the model itself. If these questions cannot be answered clearly, do not keep adding “more autonomous” features.

Always-on agents shift competition from entry-point messaging to who can maintain a large number of isolated, recoverable, auditable cloud work environments at acceptable cost. The ratio of compute, storage, and network will change accordingly: accelerator resources remain important, but general-purpose compute, state media, egress governance, and identity systems will account for a higher share than in the chat era. Only after this allocation is calculated clearly does it make sense to discuss experience differences and ensure that infrastructure investment keeps up with product promises.