Reducing Token Costs in Long-Running LLM Workflows: Practical Strategies

    Long-running LLM workflows can become expensive because large system prompts and growing conversation history are resent on every step. This article explains where those tokens come from and how to cut cost through prompt compression, context trimming, rule-based control, caching, and model routing.

    Introduction: When General Models Meet Long-Running Workflows

    In production, large language models are often affordable for a single chat turn, yet costs can rise sharply when they drive long-running automated workflows such as periodic checks, multi-agent collaboration, or automated operations.

    The difference is architectural. A general chat model uses short request-and-response interactions with relatively small context. A long-running workflow is a continuous closed loop. A complex task may involve dozens or even hundreds of tool calls, and each step requires the model to reason and decide. Without optimization, this high-frequency, long-duration pattern can consume a large share of the budget. One developer found that an agent performing a simple status check every half hour, while producing no useful business output, could still generate substantial API charges overnight. That idle-run cost shows how easily token use can get out of control.

    Where the Token Cost Comes From

    Uncontrolled cost mainly comes from two kinds of repeated loading.

    1. Large system prompts. To define the task boundary and available tools, a system prompt is often very detailed. It may include the role, tool list, calling rules, and business background, and can easily reach 5,000 to 10,000 tokens. Under a conventional API calling pattern, every request must include the full system prompt. If a workflow makes 30 tool calls, those 10,000 tokens are sent 30 times.
    2. A snowballing conversation history. As the workflow proceeds, each reasoning step, tool request, and tool result is appended to the context. On turn N, the model must read all previous turns to stay consistent. This linearly growing history makes later calls more expensive and can approach the model’s context-window limit.

    Strategy 1: Compress and Load System Prompts in Layers

    The first step is to simplify and restructure the system prompt. Do not put every rule into one large prompt.

    • Compress and structure instructions. Rewrite the system prompt with structured formats such as Markdown or JSON, and remove redundant natural-language explanation. Turn complex operating rules into a concise standard operating procedure, keeping only the information required for decisions.
    • Load instructions in layers. Split the prompt into static base instructions and dynamic context instructions. Static instructions cover basic constraints and general tool-calling rules. Dynamic instructions are injected only when a specific step needs them. For example, add data-cleaning rules only when the agent enters the data-cleaning stage, rather than carrying them throughout the workflow.

    Strategy 2: Trim and Summarize Context History

    Unlimited history growth is a major driver of cost. The goal is to preserve logical continuity while controlling token use.

    • Use a sliding window. Keep only a fixed number of recent turns, such as the latest five, and drop earlier history. This is simple and effective, but it can lose important early decisions.
    • Summarize older history. When history exceeds a threshold, make a summary call. Use a lightweight or lower-cost model to condense earlier tool calls and results into a summary of a few hundred tokens, then replace the long history with that summary. This preserves the core context while sharply reducing input tokens on later calls.
    • Filter unneeded tool output. Tools such as log queries and web retrieval can return large volumes of raw data. Before adding a result to the context, use regular expressions or code to extract only key fields or a short summary, instead of sending long unused text to the model.

    Strategy 3: Optimize Orchestration and Apply Rules First

    LLMs are well suited to uncertain reasoning, but deterministic flow control is cheaper and more stable in ordinary code.

    • Handle clear rules before calling the LLM. Not every step needs a model decision. For explicit logic such as “if the state is A, perform operation B,” use conventional control flow, such as if-else logic or a rules engine. Call the model only when the state is ambiguous and requires deeper reasoning.
    • Avoid idle runs. For periodic checks, do not have the model analyze the full state from scratch every time. First use a low-cost script to see whether the system state has changed. If it has not, skip the LLM call for that round. Wake the model for deeper analysis and decisions only when the script detects an anomaly or a state change.

    Strategy 4: Use Caching and Split Work Across Models

    • Use context caching. Many inference frameworks and cloud services support prompt caching. Highly repeated system prompts can be marked as cacheable so the serving side can reuse computed KV cache, reducing both repeated computation latency and cost. Use this general capability where it is available.
    • Assign work to models of different sizes. A long-running workflow does not need the largest flagship model at every step. Lightweight models in the tens-of-billions-parameter range can handle simple intent recognition, parameter extraction, and history summarization. Reserve models in the hundreds-of-billions-parameter range for complex multi-step planning, code generation, and other critical points. Sensible routing can substantially reduce overall cost.

    Conclusion

    High token consumption in long-running LLM workflows is not an unavoidable gap. When system prompts reach tens of thousands of tokens and context history keeps accumulating, the solution is a systems approach: layered prompt compression, dynamic context trimming, rule-based control before model calls, and cooperation between smaller and larger models. Combining the flexibility of LLMs with the determinism of conventional engineering is what makes a long-running automated workflow both capable and economical.