Two Trillion Tokens in 30 Minutes: Enterprise AI Governance and Inference Engineering in Practice

Introduction: Three Headlines, One Question

Three pieces of AI news from this week appear to have little in common:

• Consumer side: According to media reports, an AI digital human host at a leading livestreaming studio triggered roughly 2 trillion tokens of large-model inference during a single 30-minute livestream.
• Capital side: A video-generation model team closed a 17.822 billion yuan funding round, and its post-money valuation now exceeds that of its parent platform.
• Security side: A large internet company announced internally that it will block external AI access to its core-level code repositories and internal documents. AI accounts that employees purchase themselves will no longer be able to reach the relevant internal resources.

Taken together, these signals describe three main lines along which generative AI is entering production systems: scaling service delivery, committing capital, and governing for security. Tokens turn the first two lines into metrics that can be measured and attributed. The third line decides how much core material an enterprise is willing to hand to a model in the first place.

Core argument: A token is both a unit of measurement and a risk boundary. The question for enterprises has shifted from “can we use AI?” to “within what boundary, at what scale, and at what unit cost can we use AI well?”

1. Understanding Tokens: The Traffic Unit of AI Services

1.1 What a Token Is

A token is the basic unit in which a large language model processes text. A tokenizer splits text into tokens according to statistical patterns of the language, so a token can be thought of as a “character or word chunk as the model sees it.” In Chinese, one common character typically maps to about one or two tokens. In code, symbols, indentation, and variable names also affect how text is split.

Models meter both input and output in tokens. For callers, tokens convert directly into cost, latency, and throughput pressure. For platform operators, they determine how much accelerator capacity is consumed. Estimating “how many tokens a half-hour consumes” is therefore, in essence, a capacity-planning exercise.

1.2 What 2 Trillion Tokens Means: A Dimensional Breakdown

The following is an order-of-magnitude estimate, assuming half an hour is 1,800 seconds:

• 2×10¹² tokens ÷ 1,800 seconds ≈ 1.11 billion tokens per second
• If an average request consumes about 1,500 tokens (1,000 input and 500 output), that works out to roughly 740,000 requests per second.
• If the interaction leans toward long-context, multi-turn conversations averaging about 30,000 tokens per request, the figure is roughly 37,000 requests per second.

These numbers show that the bottleneck on the inference side is not determined by “how many users there are.” It is determined by how long the context is and how long the output is for each request. Livestream traffic is often bursty and puts three concrete requirements on the engineering architecture:

1. Elastic scaling: Detect traffic spikes within seconds and spin up inference instances quickly. Release resources promptly once traffic falls, so you do not pay for peak capacity year-round.
2. Tiered service: Use SLAs to separate real-time interactive requests from offline batch jobs. When resources are tight, prioritize latency-sensitive requests and move generative tasks to a message queue for asynchronous processing.
3. Intelligent scheduling: Route requests to different instances based on estimated input and output length. This keeps a long request from blocking an entire batch of short ones (head-of-line blocking).

1.3 Profiling Token Traffic: Four Dimensions That Reveal Business Characteristics

Dimension	Typical observations	Corresponding optimization
Input length	Context window, retrieved knowledge-base snippets, conversation history	Context compression, reranking of retrieved results, prefix reuse
Output length	Length of the chain of thought, generation mode, max-output settings	Cap generation length, pick a model per task
Request pattern	Bursty vs. steady, peak-to-trough ratio	Elastic scaling, peak shaving, asynchronous processing
Calling party	Human interaction, automated agent calls, batch jobs	Per-caller quotas, task prioritization

Agent applications deserve special attention on the calling-party dimension. An agent that can “think → call a tool → think again” may trigger several model calls in a single task, and its actual token consumption can be several times to tens of times that of an ordinary conversation. Without a per-call budget, a single runaway task can burn through an entire team’s monthly allocation.

2. Inference Engineering: Turning “2 Trillion” into Controllable Cost and Latency

2.1 Two Metrics to Keep Separate: TTFT and TPOT

When evaluating inference performance, it helps to separate two metrics:

• TTFT (time to first token): the time from sending a request to receiving the first token. It determines how fast the system feels to users and is largely determined by the prefill stage.
• TPOT (time per output token): the average time to generate each subsequent token. It determines how smoothly output streams and is mainly affected by memory bandwidth and batch size.

Livestream-style scenarios are sensitive to both. A slightly slow first token makes users feel the system has stalled. Choppy output throws the digital human’s lip sync out of alignment with its speech. Capacity planning should work backward from these two metrics as constraints, rather than simply maximizing total throughput.

2.2 Key Mechanisms in Inference Architecture

Mainstream inference services in the industry typically use the mechanisms below. Understanding them helps you judge whether an inference platform is “good enough”:

1. Continuous batching: New requests are inserted dynamically without waiting for every request in the current batch to finish, which significantly improves accelerator utilization.
2. KV cache reuse: An identical prefix (such as the same system prompt or the same product knowledge) is computed only once and reused by subsequent requests. This can substantially cut the compute spent on repeated input.
3. Disaggregated deployment: The compute-intensive prefill stage and the memory-access-intensive decode stage run in separate resource pools, each scaling independently.
4. Quantization and sparsity: Low-precision computation and Mixture-of-Experts (MoE) architectures reduce the compute required per inference.

2.3 An Engineering Optimization Checklist

• Tiered model routing: Send simple tasks such as intent recognition and text classification to small models, and call large models only for complex reasoning. In practice, a large share of production traffic does not need a large model at all.
• Context slimming: Rerank and truncate retrieved knowledge-base results, summarize and compress multi-turn history, and keep only content relevant to the current question.
• Semantic caching: Cache results for frequently repeated questions and return them directly on a hit. Pay strict attention to freshness, and never share cached results across users with different permissions.
• Streaming output: Return tokens as they are generated, trading “time to completion” for “perceived latency” and improving the interaction experience.
• Capacity load testing: Load test with request-length distributions that resemble real traffic, not with short texts only.

3. Why Restrict External AI Tools: Where the Risks Come From

3.1 Key Points of the Announcement

Based on publicly available information, the policy covers the following:

1. What is controlled: external general-purpose AI tools that employees access through personal accounts;
2. Scope: core-level code repositories and internal documents, with the block taking effect from the month the policy comes into force;
3. How it is enforced: external accounts can no longer reach the relevant internal resources, which is a technical block applied at the resource side.

3.2 A Breakdown of Risk Sources

• Intellectual property leakage: Source code is one of the most valuable assets of a software company. Pasting core modules into an external tool effectively takes the code outside the enterprise’s controllable boundary, and where it goes afterwards is hard to trace.
• Personal information and commercially sensitive data: Internal documents often contain personal information as well as unpublished financial and customer information. Once content leaves the enterprise environment, it is governed by the external platform’s terms and hosting locations, which can create compliance risk.
• Supply chain security: Code generated by external models may contain known vulnerabilities or snippets whose licenses are incompatible with the project. The former widens the attack surface; the latter creates the risk of legal disputes.
• Compliance and audit requirements: Certain entities, such as operators of critical information infrastructure, have national security review obligations when procuring network products and services. Data processing in industries such as finance and healthcare must also meet sector-specific regulatory requirements.
• Shadow AI and unauditability: Tools that employees subscribe to personally sit outside the enterprise’s procurement, monitoring, and audit systems. When an incident occurs, the enterprise can neither trace where the data went nor demonstrate that it fulfilled its management obligations.

3.3 Blocking and Enabling Must Go Together

If you only block without providing an alternative, employees tend to switch to more covert methods, and the risk becomes even less controllable. Mature approaches generally follow three principles:

1. Classify by data, not by tool: Public materials and core code call for different policies.
2. Replace personal accounts with a unified enterprise entry point: Give employees something that is both usable and trustworthy.
3. Make everything auditable: Every call can be traced to a person, a repository, and a point in time.

4. Designing an Internal AI Access Governance System

4.1 Overall Architecture

The foundation of a governance system is “one gateway, three control layers, one closed loop”:

• One gateway: all AI calls pass through a unified enterprise AI gateway;
• Three control layers: identity, data, and content;
• One closed loop: monitor → alert → respond → review.

Developer / employee endpoints
        │
        ▼
┌──────────────────────────────────────────┐
│ Unified identity authentication +        │
│ zero-trust access control                │
└────────────────────┬─────────────────────┘
                     ▼
┌──────────────────────────────────────────┐
│ Enterprise AI Gateway                    │
│ AuthN → data classification → masking    │
│ → content review → routing → metering    │
└──────┬──────────────────────────┬────────┘
       ▼                          ▼
Internal model services    Approved external
                           model services
       │                          │
       └────────────┬─────────────┘
                    ▼
   Audit logs / monitoring & alerts / cost attribution

The gateway can be deployed on a container platform and scaled elastically with traffic. Audit logs and call metrics are written to a logging service and a monitoring and alerting service. Raw prompts and completions that need to be retained long term can be stored in object storage with access policies set per classification level. On cloud platforms such as Baidu AI Cloud, these are all general-purpose platform capabilities. Specific choices should be evaluated against your own technology stack.

4.2 Data Classification: Where Governance Starts

A four-tier scheme commonly used in the industry is a reasonable starting point:

Level	Definition	Examples	Policy for AI tools
L1 Public	Can be published externally	Website copy, public documentation	Approved external tools are allowed
L2 Internal	For internal awareness only	General business plans, meeting minutes	Prefer internal enterprise model services
L3 Confidential	A leak would cause significant loss	Unpublished product plans, customer lists	Internal services only, with mandatory masking
L4 Core	A leak would cause major loss or cross a compliance red line	Core algorithms, keys and credentials, sensitive personal information	General-purpose LLMs are prohibited; use specialized capabilities or manual processes instead

Key points for putting code classification into practice:

• Tag the classification in metadata, and refine the rules by directory and file extension (for example, treat key and credential config files as L4 by default);
• Sync classification results to the unified identity system so they can serve as a basis for access control;
• Review classifications periodically to prevent “upgrade-only” drift and keep the scheme from becoming rigid;
• Classification does not end with tagging. You also need differentiated technical policies: close all external access channels based on personal accounts for L4 repositories, and enforce masking for L2 and L3 repositories.

4.3 A Policy Engine: Making Rules Executable and Auditable

Writing classification rules as code keeps policies from being scattered across multiple systems. The YAML below is an illustrative configuration that shows the structure of a policy. Its fields are descriptive placeholders and do not represent any specific product interface.

# Illustrative policy configuration. Fields are descriptive
# placeholders and do not represent any specific product interface.
version: 1
defaults:
  default_action: allow
  logging: metadata_only
rules:
  - id: block-external-tools-on-core-repo
    description: Block external AI tools from accessing core-level repositories
    when:
      repo.level: L4
      tool.type: external
    then:
      action: deny
      reason: CORE_REPO_EXTERNAL_TOOL_BLOCKED
      alert: security-oncall
      log_level: full_metadata
  - id: mask-pii-in-internal-docs
    description: Mask personal information before internal documents reach a model
    when:
      data.level: [L3, L4]
      action: send_to_model
    then:
      action: transform
      transforms: [mask_pii, mask_secret_like]
      require_approval: true
      retention_days: 30
  - id: budget-guard-for-agents
    description: Token budget cap for a single agent task
    when:
      caller.type: agent
    then:
      action: throttle
      max_tokens_per_task: 200000
      on_exceed: pause_and_notify

4.4 Content Masking and Detection

Before content reaches a model, the gateway should perform the following processing:

1. Regex matching: Detect structured sensitive information such as national ID numbers, bank card numbers, email addresses, and phone numbers;
2. Named entity recognition: Detect semi-structured information such as names, addresses, and organizations;
3. Code and secret detection: Detect strings that look like credentials, as well as internal system names and addresses that appear in code comments;
4. Document fingerprinting: Compare fingerprints of chunks from confidential documents against the input to determine whether it comes from a restricted document.

When masking, a good approach is “stable placeholders plus a reversible mapping table.” Mapping the same entity to the same placeholder within a session helps the model keep track of context, and the mapping table stays on the enterprise side. Note, however, that if the mapping is sent to the model along with the context, the model can in theory still reconstruct the sensitive information. For L4 content, outright blocking is therefore the safer option than masking and letting it through.

4.5 Discovering Shadow AI and Providing Alternatives

How to discover it:

• Aggregate traffic at enterprise network egress points to identify endpoints connecting to unknown AI domains;
• Use application inventories from endpoint management software to track installations of personal AI subscription tools;
• Scan code for traces such as “generated by AI” in comments and commit messages.

Providing alternatives: Once risks are identified, employees need a compliant and usable alternative. This includes an enterprise-wide AI assistant, a code assistance service, or privately deployed models. It also means clearly communicating the boundaries of data use, such as whether input content is used for model training, how long logs are retained, and who can view them. Transparency itself is an effective way to reduce the incentive to violate policy.

4.6 Metering, Auditing, and Cost Operations

Every call should produce a structured audit record. Example fields:

{
  "request_id": "r-20251106-0001",
  "timestamp": "2025-11-06T02:00:00Z",
  "user": "u_10086",
  "team": "growth-eng",
  "channel": "internal-assistant",
  "data_level": "L4",
  "repo_id": "repo_core_001",
  "tool_type": "internal",
  "action": "blocked",
  "policy_id": "block-external-tools-on-core-repo",
  "model": "internal-model-a",
  "tokens": { "input": 0, "output": 0 },
  "latency_ms": 12
}

Audit logs have to serve two needs at once:

• Security auditing: Records should not contain raw text. When necessary, retain only masked text or hashes. Retention periods and access permissions must meet compliance requirements.
• Cost operations: Aggregate token consumption across three dimensions: team, application, and model. This produces comparable unit-cost metrics such as “cost per 1,000 tokens.”

The code below is an illustration of token budget control on the gateway side:

def handle_request(req, quota_store, usage_sink):
    # 1. Identity and classification check
    level = classify_repo(req.repo_id)          # L1 ~ L4
    if req.tool_type == "external" and level == "L4":
        quota_store.block(req.user, reason="L4_EXTERNAL_BLOCKED")
        return {"action": "blocked", "reason": "L4_EXTERNAL_BLOCKED"}
    # 2. Content masking (applied only to content allowed to reach the model)
    safe_text, mapping = desensitize(req.prompt, max_level=level)
    # 3. Token budget check (estimate + reserve)
    estimated = estimate_tokens(safe_text) * 1.2
    if not quota_store.try_reserve(req.user, estimated):
        return {"action": "throttled", "reason": "TOKEN_BUDGET_EXCEEDED"}
    # 4. Model routing: prefer a small model, escalate when necessary
    model = route_model(req.task_type, level)
    result = model.generate(safe_text)
    # 5. Metering and auditing
    usage = {"input": result.usage.prompt_tokens,
             "output": result.usage.completion_tokens}
    quota_store.commit(req.user, usage)
    usage_sink.write(build_audit_record(req, level, model, usage, "allowed"))
    return {"action": "allowed", "result": result.text}

4.7 Organizational Mechanisms: Embedding Technical Controls into Processes

• Unified policy: Define which tools each data level may use, and where the red line is for personal accounts.
• Clear approval path: For exceptions that genuinely require an external tool, provide a fast-track approval channel with time-limited authorization and a retained record.
• Ongoing training: Explaining with real cases “what happens when you paste a piece of code” is more effective than issuing bans alone.
• Metrics tracking: Monitor the number of policy-violating access attempts, the masking hit rate, the false positive rate, average response time, and employee satisfaction with alternative tools.

5. Rollout Roadmap: Three Phases Over 90 Days

1. Days 1–30: Inventory and Classification
   • Deliverables: a classified inventory of data and code assets, and a report on current external AI tool usage
   • Acceptance criteria: 100% of core-level repositories are tagged with a classification
2. Days 31–60: Pilot and Enforcement
   • Deliverables: an AI gateway pilot, a policy engine, audit logging, and a masking service
   • Acceptance criteria: 100% of requests from external tools to core resources within the pilot scope are blocked, with a false positive rate below an agreed threshold
3. Days 61–90: Full Rollout and Cost Operations
   • Deliverables: a company-wide unified entry point, a token metering dashboard, budget alerts, and an exception approval process
   • Acceptance criteria: zero unauthorized data egress involving core assets, and a comparable, continuously declining unit cost per token

6. Conclusion

Two trillion tokens in half an hour measures the upper bound of large-scale AI service capability. The 17.822 billion yuan raise and a valuation above the parent platform reflect capital’s long-term expectations for the video-generation space. A large internet company tightening access to internal material is the governance choice enterprises inevitably make once AI shifts from “tool” to “infrastructure.”

All three point to the same conclusion: competitiveness in the AI era depends not only on how strong the model is, but on whether the engineering system can keep tokens accounted for, accurately measured, and well protected. A practical place to start is with three steps:

1. Classify your data;
2. Build a unified entry point;
3. Connect token metering to cost operations.

Done this way, enterprises can keep releasing the productivity of generative AI on a foundation that is secure and under control.