Cost-Efficient Refactoring of High-Concurrency LLM Orchestration Systems

1. From “Calling a Model” to “Operating an Inference System”

In the early days of LLM applications, teams could often get intelligent Q&A, content generation, ticket assistance, and knowledge retrieval into production by simply integrating a model, writing prompts, and wiring together business flows. But once request volume keeps growing and the system needs to process billions of tokens per day, the core question is no longer “can the model answer?” but “how do we keep delivering answers at controllable cost, stable latency, and in an auditable way?”

One large telecommunications industry practice disclosed that after its LLM workload reached roughly 8 billion tokens per day, it had to rebuild its existing AI orchestration system and systematically govern it to significantly reduce costs. This case reflects a general rule: inference cost does not grow only linearly with the number of calls. Context bloat, duplicate requests, wasted retries, inappropriate model selection, peak/off-peak traffic differences, and redundant pipeline orchestration can all cause costs to spiral out of control at scale.

Therefore, enterprises need to treat LLM services as an “inference production system.” Such a system includes not only the model itself but also the request entry point, routing decisions, context construction, knowledge retrieval, caching layer, task queues, monitoring and alerting, quality evaluation, and cost accounting.

---

2. Typical Challenges of Processing Tokens at Massive Scale

2.1 Cost Amplification from Context Length

Token consumption typically consists of input and output. The input side often includes system instructions, conversation history, user questions, retrieved documents, and tool return results; the output side depends on answer length, reasoning steps, and structured format requirements.

If every request carries the full history, or if the retrieval module returns large amounts of unfiltered documents, input tokens will rapidly expand. Especially in multi-turn conversations, a user question originally only a few hundred words may expand dozens of times due to redundant context.

Common problems include:

• Conversation history accumulating without bounds, with no summarization or trimming;
• Retrieval results concatenated by quantity rather than filtered by relevance;
• Multiple orchestration nodes repeatedly passing the same background information;
• Tool call results fed back to the model without compression;
• Overly long system prompts used in pursuit of “covering every possibility.”

These problems may seem like prompt details, but in practice they directly determine the unit cost of the inference system.

2.2 Mismatch Between Model Capability and Task Difficulty

Not all tasks require the highest-capability model. A large number of requests may simply involve classification, extraction, rewriting, format validation, intent recognition, short-text summarization, or rule completion. If all these lightweight tasks flow to high-cost models, the waste of resources is obvious.

Model selection should be based on task complexity, accuracy targets, response latency, and risk level, rather than a single criterion of “best performance.” A reasonable system usually includes multiple capability tiers:

• Rule and template tier: handles deterministic, fixed-format tasks;
• Lightweight model tier: handles standard tasks such as classification, extraction, and short-text generation;
• General model tier: handles routine dialogue, knowledge Q&A, and content organization;
• High-capability model tier: handles complex analysis, multi-step planning, and high-risk decision support.

The core principle: give every request the lowest-cost capability that satisfies quality requirements.

2.3 Peak Traffic and Synchronous Pipeline Pressure

Enterprise internal assistants, customer service systems, and operations platforms often have obvious access peaks. If all tasks use synchronous calls, peaks will trigger a chain reaction of queuing, timeouts, amplified retries, and resource contention.

For example, after a user initiates a request, if the system simultaneously performs knowledge retrieval, content moderation, multiple tool calls, and multiple model generations, then any slow stage can block the entire pipeline. For tasks that do not require immediate return—such as batch summarization, document tag generation, offline quality checks, and daily report drafting—asynchronous queues are more appropriate.

---

3. Core Principles for Refactoring the AI Orchestration System

3.1 Establish a Unified Request Entry Point First

LLM calls should not be initiated directly and dispersedly from various business services. Dispersed calls make version management difficult, prevent cost aggregation, preclude unified policies, and complicate security auditing and troubleshooting.

It is recommended to build a unified AI access layer responsible for:

• Unified identity authentication and permission checks;
• Unified request format and result format;
• Recording the model, version, caller, and token usage;
• Enforcing rate limiting, circuit breaking, timeouts, and retry policies;
• Executing routing and degradation based on business tags;
• Performing security handling on sensitive inputs and outputs;
• Providing a complete data foundation for subsequent cost analysis.

A request can be abstracted as follows:

{
  "request_id": "unique request identifier",
  "scene": "business scenario identifier",
  "priority": "high",
  "input": {
    "query": "user question",
    "context": "trimmed context"
  },
  "constraints": {
    "max_output_length": 800,
    "latency_target_ms": 3000,
    "quality_level": "standard"
  }
}

The point here is not the field names but making the orchestration system clearly aware of: which scenario the request belongs to, its priority, acceptable latency, and whether output length is constrained. Without this information, the system can only adopt a one-size-fits-all high-cost strategy.

3.2 Replace Fixed Model Binding with Routing Policies

Binding to a single fixed model is the most common inefficiency in operating at scale. A more reasonable approach is dynamic routing based on policies.

A basic routing logic can consider the following dimensions:

• Task type: Q&A, summarization, classification, extraction, generation, moderation;
• Input length: short input, medium context, extra-long documents;
• Output requirements: short answer, structured result, long-form content;
• Quality level: normal, important, high-risk;
• Latency requirement: real-time, near-real-time, offline;
• Historical performance: success rate, user feedback, and manual revision rate for this type of request.

Illustrative pseudocode:

def route_request(task, input_size, quality_level, realtime):
    if task in ["classification", "extraction"] and input_size < 2000:
        return "lightweight_path"
    if not realtime and task in ["batch_summary", "document_tagging"]:
        return "async_batch_path"
    if quality_level == "high" or input_size > 20000:
        return "high_capability_path"
    return "general_path"

In actual engineering, routing policies should also support canary releases. No new policy should directly cover all traffic; it should first be validated on a small percentage of requests for quality, latency, and cost changes.

---

4. Key Techniques for Controlling Token Consumption

4.1 Context Compression and Session Layering

More context is not always better. What the model actually needs is information directly relevant to the current question, not the complete log.

A “three-layer session memory” mechanism can be adopted:

• Short-term memory: retain the most recent few turns of raw conversation to maintain current semantics;
• Summary memory: compress earlier conversation into a structured summary, retaining key facts, preferences, and to-dos;
• Long-term memory: persist reusable information into a knowledge base or user profile, retrieved only when needed.

For example, if a user has conducted twenty consecutive turns of dialogue, the system does not need to put all history verbatim into the next request; it can retain the most recent three to five turns of original text and append a verified summary. This maintains continuity while significantly reducing input tokens.

4.2 Reranking and Truncation of Retrieval Results

Knowledge retrieval is an important part of enterprise LLM applications, but it is also a major source of context redundancy. If large numbers of document fragments are passed to the model every time, not only does cost increase, but answer accuracy may decrease because irrelevant content interferes with model judgment.

The following process is recommended:

1. Recall candidate documents based on the question;
2. Rerank candidate content by relevance;
3. Remove duplicate fragments and low-information-density fragments;
4. Select the most valuable content according to the token budget;
5. For overly long fragments, first produce a local summary, then provide it to the generation model.

A context budget should be set for each business scenario. For example, real-time customer service can set a tighter retrieval budget, while analysis report generation can allow a larger budget—but the value of each piece of material should still be evaluated.

4.3 Caching Duplicate Results and Intermediate Results

In high-frequency business, many requests are repetitive or highly similar—for example, common policy Q&A, operation instructions, fixed-format rewriting, and popular question summaries. For these requests, multi-level caching can be used:

• Exact cache: return existing results directly for identical inputs;
• Semantic cache: reuse results when questions are phrased differently but have the same intent;
• Retrieval cache: reuse retrieval results for the same topic;
• Tool result cache: set a validity period for data queries that change infrequently;
• Prompt fragment cache: reuse stable system instructions and common context.

Note that caches must have invalidation mechanisms and be linked to knowledge updates, permission changes, and business state changes. For content that is highly time-sensitive or involves individual data, old results cannot simply be reused.

4.4 Control Output Length and Stop Conditions

Much cost waste in many applications comes from unnecessary long outputs. For tasks such as classification, extraction, and field completion, the model only needs to return a short structured result; for Q&A scenarios, a reasonable upper limit should also be set based on page capacity and user reading habits.

Redundancy can be reduced through explicit constraints:

• Require the conclusion first, then necessary explanation;
• Limit the number of fields for structured tasks;
• Set a maximum number of items for list-type results;
• Execute long-form generation in segments, allowing users to continue on demand;
• Design “detailed explanation” as a secondary request actively triggered by the user.

This is not simply compressing answers but directing output resources toward genuinely valuable information.

---

5. Synchronous and Asynchronous Coordination: Avoiding Cost Runaway at Peak

Not every task must be completed instantly. Splitting tasks by interaction timeliness is an important way to improve system throughput.

Real-Time Tasks

Suitable for scenarios where the user is waiting for a result, such as online Q&A, real-time assistance, and page content recommendations. Such tasks should prioritize latency and adopt stricter timeout and degradation policies.

Near-Real-Time Tasks

Suitable for tasks that can wait seconds to tens of seconds, such as complex form generation, long-text rewriting, and synthesis of multiple sources. These can enter a short queue to avoid directly blocking the main request thread.

Offline Tasks

Suitable for batch document processing, historical data analysis, content archiving, quality sampling, and periodic report generation. These should be scheduled during off-peak periods whenever possible, and batch processing should be used to improve resource utilization.

For offline tasks, the system can also schedule by priority, deadline, and estimated token volume. This prevents low-value large tasks from crowding out resources and reserves capacity for high-priority tasks.

---

6. Establish a Unified Measurement System for Quality, Cost, and Latency

Without measurement, there is no way to prove whether optimization is effective. Enterprises cannot look only at total call volume or total bills; they need fine-grained metrics.

At minimum, the following metrics should be tracked:

Metric Category	Key Metrics	Purpose
Cost metrics	Tokens per request, cost per task, cache hit rate	Locate sources of resource waste
Performance metrics	Average latency, percentile latency, queue time, timeout rate	Assess user experience and capacity
Quality metrics	Task success rate, manual revision rate, user satisfaction	Prevent cost reduction from harming quality
Stability metrics	Retry rate, failure rate, degradation rate, queue backlog	Identify pipeline bottlenecks
Governance metrics	Scenario coverage, caller distribution, abnormal usage	Support auditing and budget management

In particular, avoid treating “token reduction” as the sole optimization goal. If context compression raises the answer error rate, or model degradation increases manual review volume, the ultimate business cost may actually be higher.

A more reasonable evaluation approach is to establish the cost per effective task:

cost per effective task
= total inference cost / number of successful tasks meeting quality standards

This metric measures quality and cost from the same business perspective.

---

7. Phased Implementation Recommendations

Phase 1: Observability

First, uniformly record request volume, input/output tokens, model paths, latency, error codes, retry counts, and business scenarios. Without baseline data, it is impossible to accurately determine which types of requests waste the most resources.

Phase 2: Identify High-Cost Scenarios

Sort by business scenario, caller, task type, and context length to find the requests with the highest token consumption, highest duplication rate, and highest failure rate. Usually, a small number of scenarios contribute most of the cost.

Phase 3: Prioritize Low-Risk Optimizations

Prioritize items that do not affect core quality—for example, limiting unnecessary output, trimming duplicate context, adding result caching, cleaning up wasted retries, and moving batch tasks to asynchronous execution.

Phase 4: Introduce Dynamic Routing and Quality Evaluation

Once sufficient samples are available, establish routing policies for different tasks and continuously calibrate them through sampled evaluation, manual review, and user feedback. For high-risk scenarios, stricter quality thresholds and human fallback should be retained.

Phase 5: Form a Continuous Operations Mechanism

AI orchestration optimization is not a one-time project. Model capabilities, business traffic, knowledge content, and user behavior all change, so token structure, cost trends, quality changes, and abnormal usage should be reviewed regularly.

---

8. Conclusion

When LLM usage reaches the scale of billions of tokens, what truly determines a system’s competitiveness is not just model capability but whether the enterprise has refined orchestration and operations capabilities. Through a unified entry point, dynamic routing, context governance, cache reuse, asynchronous scheduling, and end-to-end measurement, teams can turn previously uncontrollable inference spending into an engineering problem that can be analyzed, optimized, and governed.

The essence of cost reduction is not simply reducing model calls but completing processing with the right task, at the right time, with the right context, and at the right capability tier. Only by safeguarding quality, latency, and security boundaries simultaneously can AI systems move from experimental applications to stable, sustainable production-grade services.