Million-Token Context: Model and Cost Selection for AI Agents

    As context windows reach one million tokens, model competition shifts from how much a model can read to how much it can actually complete. This article explains why effective context matters more than raw window size, how to read benchmark scores without over-interpreting noise, and when prompt caching pays off — with example pricing, cost formulas, and runnable code. It also provides a long-horizon agent skeleton with external progress and failure tracing, practical guidance for terminal tasks, CAD scripting, financial document workflows, and hallucination control, plus a selection decision table, pre-launch validation metrics, an FAQ, and an action checklist.

    1. Bottom Line First: Million-Token Context Changes the Task Boundary

    When the context window reaches one million tokens, model competition shifts from “can it read more?” to “can it finish the job?” For engineering teams, the most important thing is not to fixate on a single leaderboard score, but to clarify three things first.

    1. The window becomes a workspace
      A million tokens is not just about fitting longer documents; it means code repositories, logs, spreadsheets, and multi-turn agent histories can live in the same workspace. But whether a task actually gets done depends on effective context, retrieval quality, state management, and cost structure.

    2. Evaluation shifts from answering to completing
      Terminal tasks, long-horizon code changes, tool calls, document generation, and result verification all test multi-step execution. A model must not only “talk the talk” but also call tools, read feedback, correct errors, and deliver results.

    3. Metrics must share a common definition first
      Percentages, 100-point scales, pass@k, internal benchmarks, and public benchmarks cannot be mixed. Undisclosed figures should not be filled in with estimates; close scores may simply be measurement noise. Selection should be based on success rate, latency, and cost per task in your own evaluation set.

    Note on data: The prices, leaderboard positions, benchmarks, and model capabilities in this article are organized from public materials and example figures. Before making a formal selection, verify specific model names, release dates, test sets, and prices against the vendor’s release pages, pricing pages, and third-party evaluation pages.

    2. Where Long Context Comes From and the “Effective Context” Trap

    Supporting a million tokens usually relies on three pieces of foundational engineering.

    1. Position encoding extrapolation
      Mainstream models commonly use rotary position embeddings. Beyond the native training length, you need interpolation, frequency scaling, or other extrapolation methods to maintain long-range position awareness; otherwise the model’s understanding of very-long-range dependencies degrades.

    2. Long-sequence training
      Training only on short text and then forcing a larger window rarely yields stable long-range understanding. Usually you introduce long documents and long code repositories in the later stages of pretraining and gradually extend the training window.

    3. Attention efficiency
      The compute for standard attention grows with the square of sequence length, and KV-cache usage grows linearly with length. Supporting a million-token window requires operator optimization, memory management, cache management, and even sparse or hierarchical attention.

    But a large window does not mean strong effective context. A common industry problem is “lost in the middle”: information at the beginning and end is more easily used, while key information in the middle is often missed. As a result, constraints in the middle of a long document, cross-file function definitions, and early decisions in a long conversation may be ignored by the model.

    Mitigation approaches include:

    • Retrieval and summarization: retrieve relevant snippets first, then expand them on demand;
    • Structured organization: add clear sections, file, and role markers to the input;
    • Redundancy for key information: place hard constraints at both the beginning and the end, and restate them periodically throughout long workflows.

    This is also why “why can’t effective context be judged by window size alone” becomes a key question in selection: the window is only a ceiling. What truly determines task success is whether information is accurately retrieved, organized, and referenced.

    3. How to Read Benchmark Scores: Unify Units First, Then Judge Noise

    Under public reporting, several kinds of metrics can be observed. For ease of understanding, the table below still uses anonymized figures and is only meant to illustrate how to read it.

    Metric This release Previous generation Peer closed-source model A Peer closed-source model B
    Terminal-constrained task pass rate (%) 65.7% — 58.2% 66.4%
    Long-horizon code benchmark pass rate (%, vendor internal) 61.2% 55.95% — —
    Tool-assisted CAD modeling pass rate (%) 92.7% — 95.9% 96.2%
    Office document QA benchmark (points, 100-point scale) 64.4 — — —
    Hallucination stress test, single-item factual error rate (%, lower is better) 8.3% 23.4% — —
    Development task self-eval pass@3 pass rate (%) 81.0% — 90.5% unfiltered —

    Three principles apply when reading the table.

    1. Do not over-interpret differences within measurement noise
      On terminal tasks, this release leads model A by about 7.5 percentage points but differs from model B by only 0.7 points. Sampling parameters, tool configurations, and runtime environments all affect results; a fraction of a point should not be read directly as a generational capability gap.

    2. Internal benchmarks only show trends
      The long-horizon code benchmark rising from 55.95% to 61.2% can indicate generational progress, but it cannot replace independent reproduction. What is truly reliable is making multi-file changes in your own repository, dependency system, and build pipeline and passing the tests.

    3. Different dimensions cannot be extrapolated from one another
      A high CAD score does not mean the same strength on code tasks; a high document QA score does not mean the same long-horizon agent capability. Selection must be broken down and evaluated per business scenario.

    4. Cost Estimation for Million-Token Context: When Does Caching Pay Off

    The example prices below are illustrative figures in cost per million tokens.

    Model Input Output Cache hit Cache write (5 min) Cache write (1 hour)
    Standard 13.00 68.00 1.30 13.00 26.00
    High-speed 26.00 136.00 1.30 26.00 52.00

    4.1 The correct formula: how much caching saves vs. no caching

    Let:

    • (W): number of tokens written to cache;
    • (H): cumulative cache-hit tokens over the cache lifetime;
    • (c_{in}): standard input price;
    • (c_{hit}): cache-hit price;
    • (c_{write}): cache-write price.

    Without caching, these input and hit contents are all billed at the standard input rate, so the cost is:

    [
    (H+W)c_{in}
    ]

    With caching, the cost is:

    [
    Hc{hit}+Wc{write}
    ]

    So the savings are:

    [
    savings=(H+W)c{in}-(Hc{hit}+Wc_{write})
    ]

    Rearranged:

    [
    savings=H(c{in}-c{hit})+W(c{in}-c{write})
    ]

    4.2 Judging the 5-minute vs. 1-hour tier

    Using the standard tier prices as an example, (c{in}=13), (c{hit}=1.3).

    5-minute tier: (c_{write}=13), equal to the input price.
    Savings:

    [
    H(13-1.3)+W(13-13)=11.7H
    ]

    That is, as long as the cached prefix produces cumulative hits afterward, (H>0), you save money; if you only write and never read, (H=0), then it is neither a gain nor a loss, but there is no extra saving either.

    1-hour tier: (c_{write}=26), twice the input price.
    Savings:

    [
    H(13-1.3)+W(13-26)=11.7H-13W
    ]

    To save money, you need:

    [
    11.7H>13W
    ]

    That is:

    [
    H>\frac{13}{11.7}W\approx1.111W
    ]

    If each request reuses the same prefix of length (W), after writing you need at least two subsequent hits — which usually means at least three total requests for the 1-hour tier to pay off.

    4.3 A worked example you can recompute

    Assume:

    • tokens written to cache (W=100000);
    • cumulative cache-hit tokens over the lifetime (H=200000).

    Savings on the 1-hour tier:

    [
    200000\times\frac{13-1.3}{1000000}-100000\times\frac{13}{1000000}
    =2.34-1.30=1.04
    ]

    The savings are positive, which means the 1-hour cache tier pays off under this assumption.

    4.4 Runnable cost-estimation code

    The code below depends only on the standard library and uses formatted output. Because floating-point representation may differ slightly, output details can vary a little between Python versions, but after rounding to two decimals the results should match.

    1. # Prices are in cost per million tokens
    2. PRICING = {
    3. "standard": {
    4. "input": 13.00,
    5. "output": 68.00,
    6. "cache_hit": 1.30,
    7. "cache_write_5m": 13.00,
    8. "cache_write_1h": 26.00,
    9. },
    10. "highspeed": {
    11. "input": 26.00,
    12. "output": 136.00,
    13. "cache_hit": 1.30,
    14. "cache_write_5m": 26.00,
    15. "cache_write_1h": 52.00,
    16. },
    17. }
    18. def estimate_cost(
    19. tier,
    20. input_tokens,
    21. output_tokens,
    22. cache_hit_tokens=0,
    23. cache_write_tokens=0,
    24. write_ttl="5m",
    25. ):
    26. if tier not in PRICING:
    27. raise ValueError(f"unknown tier: {tier}")
    28. if write_ttl not in ("5m", "1h"):
    29. raise ValueError("write_ttl must be '5m' or '1h'")
    30. if min(input_tokens, output_tokens, cache_hit_tokens, cache_write_tokens) < 0:
    31. raise ValueError("token counts cannot be negative")
    32. p = PRICING[tier]
    33. billed_input = input_tokens - cache_hit_tokens - cache_write_tokens
    34. if billed_input < 0:
    35. raise ValueError("cache hits plus cache writes cannot exceed total input tokens")
    36. return (
    37. billed_input * p["input"]
    38. + cache_hit_tokens * p["cache_hit"]
    39. + cache_write_tokens * p[f"cache_write_{write_ttl}"]
    40. + output_tokens * p["output"]
    41. ) / 1_000_000
    42. if __name__ == "__main__":
    43. case = dict(
    44. input_tokens=800_000,
    45. output_tokens=50_000,
    46. cache_hit_tokens=600_000,
    47. cache_write_tokens=100_000,
    48. write_ttl="1h",
    49. )
    50. std = estimate_cost("standard", **case)
    51. fast = estimate_cost("highspeed", **case)
    52. # No caching: 800k input at 13.00, 50k output at 68.00
    53. no_cache = (800_000 * 13.00 + 50_000 * 68.00) / 1_000_000
    54. print(f"standard {std:.2f} highspeed {fast:.2f} no_cache {no_cache:.2f}")
    55. print(f"ratio {fast / std:.3f}")

    Under this example, the standard tier on the 1-hour tier is about 8.08, the standard tier on the 5-minute tier is about 6.78, the high-speed tier on the 1-hour tier is about 15.38, and no caching is about 13.80. The high-speed tier is about 1.90 times the standard tier, but this ratio depends on the cache-hit rate; the lower the hit rate, the closer the ratio approaches 2.

    5. Long-Horizon Agent Skeleton: External Progress, On-Demand Retrieval, Failure Tracing

    A long-horizon agent cannot rely only on conversational memory. A more robust architecture should include:

    • Layered context: on top of a code index or vector-retrieval service, inject relevant files on demand;
    • External task state: write plans and progress to a state file or database;
    • Verification loop: after each step executes, read the real output and verify it;
    • Trace recording: log tool calls, errors, and results for auditing and human intervention.

    Below is a skeleton example that depends only on the standard library. The code uses types such as Optional, List, and Tuple, compatible with Python 3.8+; if you switch to syntax like str | None, you would need Python 3.10+.

    1. # Long-horizon agent main loop skeleton: external progress, failure tracing
    2. # If a step still fails after reaching the max attempts, it records skipped/failure reason
    3. # and does not disguise a skip as success
    4. import json
    5. import os
    6. from dataclasses import dataclass, field, asdict
    7. from typing import Protocol, List, Tuple, Optional
    8. MAX_ATTEMPTS = 3
    9. MAX_ERR_CHARS = 2000
    10. @dataclass
    11. class Step:
    12. goal: str
    13. done: bool = False
    14. attempts: int = 0
    15. max_attempts: int = MAX_ATTEMPTS
    16. last_error: Optional[str] = None
    17. @dataclass
    18. class Action:
    19. kind: str # "read" or "run"
    20. target: str # file path or command
    21. args: List[str] = field(default_factory=list)
    22. class Model(Protocol):
    23. def plan(self, task: str) -> List[Step]: ...
    24. def decide(self, item: Step, context: str) -> Action: ...
    25. class Tools(Protocol):
    26. def search(self, goal: str, k: int = 5) -> str: ...
    27. def run(self, action: Action) -> Tuple[bool, str]: ...
    28. class PlanState:
    29. def __init__(self, task, steps):
    30. self.task = task
    31. self.steps = steps
    32. def save(self, path):
    33. tmp = path + ".tmp"
    34. with open(tmp, "w", encoding="utf-8") as f:
    35. json.dump(
    36. {"task": self.task, "steps": [asdict(s) for s in self.steps]},
    37. f,
    38. ensure_ascii=False,
    39. indent=2,
    40. )
    41. os.replace(tmp, path)
    42. def load_state(path, task, planner):
    43. if os.path.exists(path):
    44. with open(path, encoding="utf-8") as f:
    45. data = json.load(f)
    46. if data.get("task") == task:
    47. return PlanState(task, [Step(**s) for s in data["steps"]])
    48. state = PlanState(task, planner.plan(task))
    49. state.save(path)
    50. return state
    51. def run_long_horizon_agent(task, model, tools, state_path="state.json", max_steps=40):
    52. state = load_state(state_path, task, model)
    53. skipped = []
    54. for i in range(1, max_steps + 1):
    55. pending = [s for s in state.steps if not s.done]
    56. if not pending:
    57. if skipped:
    58. return {
    59. "ok": False,
    60. "reason": "some_steps_failed_or_skipped",
    61. "skipped": skipped,
    62. "steps": len(state.steps),
    63. }
    64. return {"ok": True, "steps": len(state.steps)}
    65. item = pending[0]
    66. # If the previous run already exhausted the attempts, explicitly skip here and record a trace
    67. if item.attempts >= item.max_attempts:
    68. item.done = True
    69. item.last_error = f"Exceeded max attempts, skipping step: {item.goal}"
    70. skipped.append({
    71. "goal": item.goal,
    72. "attempts": item.attempts,
    73. "last_error": item.last_error,
    74. })
    75. state.save(state_path)
    76. continue
    77. context = tools.search(item.goal, k=5)
    78. try:
    79. action = model.decide(item, context)
    80. ok, output = tools.run(action)
    81. except Exception as exc:
    82. ok, output = False, f"{type(exc).__name__}: {exc}"
    83. item.attempts += 1
    84. if ok:
    85. item.done = True
    86. item.last_error = None
    87. else:
    88. item.done = False
    89. item.last_error = output[:MAX_ERR_CHARS]
    90. # Reached max attempts and still failed: mark done, but record the failure and allow human intervention
    91. if item.attempts >= item.max_attempts:
    92. item.done = True
    93. item.last_error = f"Exceeded max attempts, skipping step: {item.last_error}"
    94. skipped.append({
    95. "goal": item.goal,
    96. "attempts": item.attempts,
    97. "last_error": item.last_error,
    98. })
    99. state.save(state_path)
    100. print(i, "OK" if ok else "ERR", item.goal, item.last_error or "")
    101. pending_left = [s.goal for s in state.steps if not s.done]
    102. return {
    103. "ok": False,
    104. "reason": "max_steps_reached",
    105. "pending": pending_left,
    106. "steps": len(state.steps),
    107. }
    108. # ---- Demo implementation: replace entirely when integrating a real project ----
    109. class DemoModel:
    110. def plan(self, task):
    111. return [
    112. Step("1: read README"),
    113. Step("2: run tests"),
    114. Step("3: generate summary"),
    115. ]
    116. def decide(self, item, context):
    117. if item.goal.startswith("1"):
    118. return Action("read", "repo/README.md")
    119. if item.goal.startswith("2"):
    120. # Intentionally fail the first time, succeed on retry
    121. if item.attempts == 0:
    122. return Action("run", "pytest --bogus-flag")
    123. return Action("run", "pytest")
    124. return Action("run", "echo", ["summary.txt", "done"])
    125. class DemoTools:
    126. def search(self, goal, k=5):
    127. try:
    128. with open("repo/README.md", encoding="utf-8") as f:
    129. return f.read()[:4000]
    130. except OSError as exc:
    131. return f"<README not found: {exc}>"
    132. def run(self, action):
    133. if action.kind == "read":
    134. with open(action.target, encoding="utf-8") as f:
    135. return True, f.read()[:4000]
    136. cmd = action.target
    137. if cmd.startswith("pytest"):
    138. if "--bogus-flag" in cmd:
    139. return False, "ERROR: unrecognized arguments: --bogus-flag"
    140. return True, "3 passed"
    141. if cmd == "echo":
    142. if len(action.args) < 2:
    143. return False, "echo needs two arguments: filename and content"
    144. path, text = action.args[0], action.args[1]
    145. with open(path, "w", encoding="utf-8") as f:
    146. f.write(text + "\n")
    147. return True, "ok"
    148. return False, f"unsupported: {cmd}"
    149. if __name__ == "__main__":
    150. os.makedirs("repo", exist_ok=True)
    151. with open("repo/README.md", "w", encoding="utf-8") as f:
    152. f.write("# demo\n")
    153. result = run_long_horizon_agent("add tests to the demo project", DemoModel(), DemoTools())
    154. print(result)

    On the first run, step 2 first returns pytest --bogus-flag and fails, then retries with pytest and succeeds. The final normal return looks like:

    1. {'ok': True, 'steps': 3}

    On a subsequent run, the program resumes from state.json; if the task is already complete, it directly returns a completed status. If a step still fails after reaching the maximum attempts, the result includes ok=False, the failed step and its reason, making human intervention easy rather than misjudging the task as fully successful.

    6. Terminal Tasks, CAD, Financial Documents, and Hallucination Control

    6.1 Terminal-constrained tasks

    Terminal tasks are closest to real development: the model can complete tasks only through the command line and file operations, the environment is usually restricted, and results must actually run. They test five stages:

    1. understand the directory structure, dependencies, and errors;
    2. break the task into a sequence of steps;
    3. call commands, read/write files, and run tests;
    4. correct direction based on output;
    5. deliver a runnable result within budget.

    For selection, you cannot look only at code generation; you must also consider tool calling, error recovery, and state-management capabilities.

    6.2 3D modeling and CAD scripting

    Tool-assisted modeling requires three layers of model capability:

    • API knowledge: understand components, coordinate systems, and lifecycle functions;
    • Geometric reasoning: convert natural-language requirements into parameterized scripts;
    • Runtime verification: run the script, inspect the result, and correct it.

    Engineering practices should include:

    • feed the model API documentation that matches the version and is scope-controlled;
    • keep generated scripts idempotent and repeatable;
    • put assets under version control;
    • run model-generated scripts only in a sandbox.

    6.3 Financial document workflows

    In financial document scenarios, large models are well suited to understanding, drafting, correlation, and narration, but key numbers should not be decided by a generative model alone. The recommended architecture is:

    1. Financial reports / research reports / ledgers
    2. → document parsing
    3. → structured data and calculation engine
    4. → large model: analysis framework, report writing
    5. → numeric consistency checks
    6. → human review
    7. → archiving and distribution

    The core principle is: separate deterministic computation from generative expression. Key numbers in a report should be traceable to the source file, page, or cell.

    6.4 Hallucination control

    In the hallucination stress test, a drop in the single-item factual error rate does not equal an equivalent rise in accuracy. If the error rate is still 8.3%, single-item accuracy is about 91.7%, which still means roughly 1 in 12 facts may be wrong. For a long-horizon agent, one error can be inherited by subsequent steps and cause the whole task to fail.

    Actionable control measures include:

    1. Retrieval grounding: require the model to answer based on retrieved source text and provide citations;
    2. Structured validation: validate fields, enumerations, and numeric ranges with a program;
    3. Tool verification: verify facts such as numbers and dates with query or calculation tools;
    4. Failure distinction: distinguish “the model did not see it” from “the model cannot do it,” then either add context or switch models accordingly.

    7. Selection Decision Table and Pre-Launch Validation

    Scenario Context need Key capability Recommended tier Cost optimization focus
    Interactive coding assistant Medium-long Code generation, multi-turn correction High-speed, latency-sensitive Stable prefix caching, limit injected code volume
    Repo-level change / long-horizon agent Long Tool calling, state management Standard, batch-oriented Inject on demand via retrieval, external progress
    Long document / research report analysis Long Long-context understanding, citation tracing Standard Cache the document prefix, chunked summarization
    Financial modeling and reporting Medium Numerical accuracy, tool calling Standard Externalize deterministic computation
    3D / CAD scripting Medium Domain API knowledge, script execution Standard Sandbox execution, failure feedback

    Before launch, it is recommended to build your own evaluation set with no fewer than 100 representative tasks, test both the standard and high-speed tiers separately, and record five metrics:

    1. success rate: task completion rate and first-pass rate;
    2. latency: end-to-end P50/P95;
    3. unit cost: cost per successful task;
    4. cache-hit rate: whether it reaches the 1-hour-tier break-even threshold;
    5. failure rollback effectiveness: whether scripts can roll back on error without polluting project files.

    To compute the cost per successful task, you can use a simple template:

    1. cost per successful task = total cost / number of successful tasks
    2. total cost = input cost + output cost + cache-hit cost + cache-write cost + failed-retry cost

    Cache usage recommendations:

    • place the system prompt, project description, and fixed documents at the very front of the request;
    • place timestamps, random numbers, and real-time data at the end to avoid busting the cache;
    • if reuse intervals are on the order of hours, consider the 1-hour tier;
    • if you cannot reach the reuse threshold, not writing the cache is more stable.

    8. FAQ: Common Questions on Selection and Cost

    Q1: Can a million-token window just hold the whole repository and all the documents?
    Not recommended. Key information in the middle may be missed, and both input and output costs rise. Prefer retrieval-based location, structured organization, and redundancy of key constraints.

    Q2: Why can’t effective context be judged by window size alone?
    Because the window is only a ceiling. Whether retrieval is accurate, whether information is clearly organized, whether key constraints are repeated, and whether the model is disturbed by noise all affect how much of that length is actually used.

    Q3: How to choose between the standard and high-speed tiers?
    For interactive, latency-sensitive scenarios, consider the high-speed tier; for batch processing, repo-level changes, and document analysis, prefer the standard tier. The high-speed tier has a higher output price but the same cache-hit price, so the decision should be based on the latency budget.

    Q4: When is caching not worth it?
    When a cache is almost never read again after being written. At current prices, the 1-hour tier needs cumulative hits to exceed about 1.11 times the written amount; the 5-minute tier saves as long as any subsequent hit occurs.

    9. Action Checklist

    1. Build an evaluation set of more than 100 real tasks with fixed scoring scripts;
    2. Run both the standard and high-speed tiers on the same evaluation set, recording success rate, P95 latency, and cost per successful task;
    3. Measure the prefix-token share and reuse count, then decide the cache tier by the formula;
    4. Design a stable prefix and place volatile content at the end;
    5. Make the long-horizon agent achieve external progress, on-demand retrieval, failure feedback, and sandbox execution;
    6. In finance, research-report, and similar scenarios, hand key calculations to deterministic tools and ensure numbers are traceable.

    The real value of million-token context is not “how much it can hold” but whether it can reliably complete tasks at a reasonable cost. In selection, effective context, caching strategy, tool verification, and safety protection should all be part of the design, rather than a single leaderboard ranking.