Scaling AI Office Assistants: Sparse Models, Token Throughput, and Cloud Capacity Planning

    This article examines how developers and cloud teams can turn large models into everyday office tools. It covers MoE sparse activation and memory sizing, pretraining and post-training balance, peak token throughput estimation, effective token measurement, infrastructure planning, open-weight production controls, and pre-launch checks. The main takeaway is to calculate activated and total parameters, memory, KV cache, and communication overhead together; estimate capacity from real request distributions; and measure product value with effective tokens rather than billing tokens alone.

    This article is intended for developers and cloud teams that need to turn large models into everyday office tools. The scope includes model architecture, inference throughput, infrastructure, and open-source deployment. The numbers in this article are calculation examples, not real data from any product. They should be replaced with your own business data.

    What MoE Sparse Activation Saves

    A dense model uses all parameters in every forward pass. As the number of parameters grows, both single-pass computation and memory reads increase. A mixture-of-experts (MoE) model splits parameters into multiple experts, and each token passes through only a small number of routed experts.

    Take a hypothetical model as an example: 200B total parameters, with only about 12B activated per forward pass. Its knowledge capacity is close to that of a large model, while its per-pass computation is close to that of a medium-sized dense model. What is saved here is computation and access to expert weights. All expert weights still need to reside somewhere, so memory footprint does not shrink accordingly.

    Can Activated Parameters Determine Memory Usage?

    No. Many teams interpret “fewer activated parameters” as “lower memory usage.” This is a common misunderstanding. Weight memory and KV cache must be calculated separately.

    First, weight memory depends on total parameter count and precision. For 200B parameters, FP16 requires about 400GB, INT8 about 200GB, and INT4 about 100GB. Even with 4-bit quantization, a single 80GB GPU cannot hold all weights. Such models usually require multi-GPU parallelism or expert parallelism.

    Second, KV cache depends on the attention structure and request shape. The KV cache size for a single batch can be estimated as follows:

    KV Cache bytes ≈ 2 × layers × KV heads × head dimension × bytes per element × sequence length × batch size

    The factor 2 represents the two caches for K and V. Grouped-query attention (GQA) reduces the number of KV heads, KV cache quantization reduces bytes per element, and paged management reduces memory fragmentation. KV cache should be controlled through these methods, not through activated parameters.

    Should Pretraining and Post-Training Scale Together?

    Usually yes. Scaling only pretraining data increases the model’s base knowledge, but it may not improve instruction following for office tasks such as document drafting, meeting notes, and code assistance. When post-training data quality is poor, the model may produce fluent answers with incorrect steps or formats that do not meet requirements.

    In practice, check three things: whether the data mix covers real office tasks, whether training difficulty increases progressively, and whether reinforcement learning rewards align with user adoption outcomes. If rewards only consider length or format, the model will learn to satisfy the scoring rule rather than solve the problem.

    How to Estimate Peak Token Throughput

    Capacity planning cannot derive conclusions directly from “10 million DAU.” It should be calculated through the following chain:

    Daily tokens ≈ DAU × requests per user × average tokens per request

    Peak token rate ≈ daily tokens × peak-period traffic share ÷ seconds in the peak period

    Assume 10 million DAU, an average of 20 requests per user per day, and an average of 3,000 tokens per request (input plus output). Daily token volume is about 600 billion. If a two-hour peak carries 20% of daily traffic, the peak rate is about 16.67 million tokens per second. This number must be replaced with your own request distribution, model speed, and hardware throughput. It is not a universal benchmark.

    Inference services must meet two types of metrics at the same time: time to first token (TTFT) and output tokens per second. The former determines the response speed users perceive, and the latter determines completion time for long answers. Target values should be defined by the product and should be evaluated using P95 and P99 percentiles, not averages alone.

    The advantage of sparse models here mainly comes from lower per-pass computation, which allows higher batching efficiency. Common methods include continuous batching, decoupling expert routing from computation with asynchronous pipelines, caching hot experts, and offloading simple requests to quantized edge nodes. These methods introduce additional communication or scheduling costs, so they should be enabled only after stress testing.

    How to Count Effective Tokens

    Billing tokens and effective tokens are not the same metric. Billing tokens are all input and output actually processed by the model. Effective tokens are the output portion adopted by users.

    An actionable definition is: effective token ratio = adopted output tokens ÷ billed output tokens. “Adopted” can be defined as the user copying, inserting into a document, or confirming execution, without retrying or deleting within a short period. At minimum, log the following fields: request ID, input token count, output token count, whether truncated, whether retried, whether adopted, and adoption action type.

    For example, if billed output tokens on a given day are 1,000 and users adopt 600 of them, the effective ratio is 60%. This number reflects product value and can also show which scenarios are wasting compute. Product and finance teams must align on definitions; otherwise, the same metric will lead to different conclusions.

    Infrastructure Planning

    Requests at the tens-of-millions DAU level are unevenly distributed by region and time, and a fixed cluster makes it difficult to balance cost and experience. A layered approach can be used:

    • Compute layer: GPU inference is deployed in containers, with autoscaling triggered by queue length and TTFT.
    • Function compute: suitable for lightweight tasks and preprocessing, but not for directly hosting large models, because cold starts that load hundreds of GB of weights take a long time.
    • Storage layer: model weights are stored in object storage, hot replicas are kept on local disks, and session state and hot data are stored in distributed cache.
    • Monitoring layer: cover token-level, request-level, and user-level metrics, with drill-down by tenant and feature module.

    Logs and billing data should be designed separately. Every token corresponds to real cost. Errors in the metering pipeline directly affect bills and optimization decisions, so regular reconciliation with billing is required.

    What Open-Source Deployments Need to Add

    Open weights lower the barrier for enterprise trials. Developers can validate scenario fit locally or in a private cluster before deciding whether to move to the cloud. Open source is only model release; for enterprises to put a model into production, supporting controls are also required.

    A minimum compliance checklist is as follows:

    Capability Key points
    Input/output filtering Block sensitive content and unauthorized instructions, and log the reason for each match
    Tenant isolation Isolate data, caches, and logs across departments or customers
    Audit logs Record caller, time, model version, and policy results; set retention according to industry compliance requirements
    Sensitive data masking Mask fields such as phone numbers and ID numbers in logs and training samples before storage
    Access control Restrict callable models, tools, and data scope by role

    Pre-Launch Checks

    Problems on peak days usually come from service availability rather than model quality. Before launch, complete at least the following:

    • Capacity stress testing: test with real request distributions and record latency percentiles and error rates.
    • Tiered rate limiting: set quotas by user tier, feature, and tenant, and return clear messages when limits are exceeded.
    • Fallback models: switch to a smaller model when the primary model is unavailable or the queue is too long, and label the result source.
    • Cache hit rate: monitor prefix cache and result cache hit rates, and trigger alerts when hit rates decline.
    • Rollback thresholds: define rollback conditions for error rate, latency, and effective token ratio in advance to avoid ad hoc discussions during incidents.

    Conclusion

    Scaling AI office assistants depends on the coordination of model architecture, inference engineering, and cloud infrastructure. Activated parameters, total parameters, memory, KV cache, and communication overhead must be calculated together; looking at only one factor leads to incorrect judgments. Teams should first estimate token throughput with real requests, then choose parallelism and quantization methods, and finally measure product value with effective tokens.

    Sparse architectures and quantization are still evolving rapidly, and today’s cost structure may not hold over the long term. Fix the estimation methods and monitoring definitions first; when models or hardware are replaced later, comparisons will be more reliable.