Million-Token LLM Selection: Evaluation, Cost, and Security

    Million-token context windows reduce the need for chunking, but they do not guarantee better results. This article explains how to test long-context retrieval, evaluate long-running agents, compare vendor metrics, estimate per-task cost with caching, and control safety and factual accuracy.

    LLM context windows have expanded from tens of thousands of tokens to millions. In theory, a model can read an entire codebase, years of financial reports, or a batch of contracts in one pass. But a larger window does not necessarily mean better results. Before procurement or deployment, enterprises need to verify three things: whether key information in long text can be found, whether long-running tasks can be completed reliably, and whether the cost per task is acceptable.

    1. What Does a Million-Token Context Window Solve?

    The direct benefit of long context is less chunking. Traditional RAG solutions split documents first and then retrieve snippets, which can easily lose relationships that span paragraphs. When a model reads complete material directly, it can perform repository-wide code review, multi-year financial report comparison, long contract clause consistency checks, and similar tasks.

    The limitations are also clear. The longer the input, the more easily information in the middle can be ignored. This phenomenon is often called “lost in the middle.” Its trigger conditions depend on task type, information position, and input length, so they need to be tested. RAG remains useful, especially when the knowledge base changes frequently, the budget is limited, or low latency is required.

    2. How Should You Test Long-Context Capability?

    The most common method is the needle-in-a-haystack test. The steps are:

    1. Divide tests into length tiers, such as 8K, 32K, 128K, 256K, and 1M tokens.
    2. Insert a known fact (the “needle”) at different positions, such as the beginning, 25%, 50%, 75%, and the end.
    3. Design three types of questions: exact extraction, cross-paragraph reasoning, and multi-file reference.
    4. Repeat each combination at least 20 times and record the accuracy.
    5. Plot the results as a heatmap to observe how accuracy changes with length and position.

    Use documents from your own business during testing, not random text. Random text cannot reflect the distracting information found in real business content.

    3. How Should Long-Running Agents Be Evaluated and Corrected?

    A long-running agent needs to maintain its goal across multiple rounds of tool calls. There are three common failure types: goal drift (moving away from the original requirement during execution), tool parameter errors (writing the wrong fields when calling an API), and state loss (forgetting previously completed steps).

    Corresponding engineering measures include:

    • Validate the returned result after each tool call. If it fails, retry once. If it still fails, stop and report.
    • Set a maximum number of steps to prevent infinite loops.
    • Save checkpoints at key points and roll back to the most recent checkpoint on failure.
    • High-risk operations such as deletion, sending, and deployment must be confirmed by a human.

    For evaluation, use 10 to 30 real tickets to form a test set, and record completion rate, average number of steps, and number of human interventions. Public leaderboards can be used as references, but you should note the benchmark name, version, and evaluation date.

    4. Which Metrics Should Enterprises Use for Selection?

    The table below lists commonly used metrics. The passing threshold should be determined by each team based on business requirements.

    Metric Why it matters Test method
    Effective context length The nominal window does not equal usable length Needle-in-a-haystack test; find the inflection point where accuracy begins to drop
    Time to first token (TTFT) Affects interactive experience Measure the same request multiple times; report the median and 95th percentile
    Output speed Affects wait time for long answers Measure output tokens per second
    Cost per task Determines the budget at scale Calculate using the formula in Section 5
    Cache hit rate Determines the actual cost of repeated context Use production logs
    Domain factual error rate Determines suitability for professional scenarios Human-labeled domain Q&A set
    Safety refusal and false refusal rates Affects both risk and usability Red-team tests and normal request samples
    Data retention and compliance terms Determines whether sensitive data can be processed Review the service agreement and confirm with legal

    Selection steps:

    1. List 3 to 5 real business tasks and prepare a gold answer for each task.
    2. Run the same inputs on candidate models and score them using the table above.
    3. If a vendor offers a high-speed version, use it for latency-sensitive tasks and use the standard version for offline batch tasks.
    4. Launch a small-traffic canary, observe for two weeks, and then expand the scope.

    5. How Should You Estimate the Cost per Task?

    Use the following formula:

    Cost per task = uncached input tokens × input price + cache-hit tokens × cache-hit price + cache-write tokens × cache-write price + output tokens × output price

    The cache-write price is usually higher than the ordinary input price, while the cache-hit price is significantly lower. Therefore, long prefixes that are reused frequently, such as system prompts or codebase background, are suitable for caching. The retention period after writing also affects cost and should be selected based on call frequency.

    Below is a hypothetical example. The numbers are only for illustrating the calculation method and do not represent any vendor’s pricing. Assume an input price of CNY 10 per million tokens, a cache-hit price of CNY 1 per million tokens, a cache-write price of CNY 12.5 per million tokens, and an output price of CNY 40 per million tokens. The system prefix is a 1-million-token codebase, with 100 calls per day, and each call adds 50K input tokens and 20K output tokens.

    • Without caching: each call has 1.05 million input tokens, costing about CNY 10.5. For 100 calls, that is about CNY 1,050. Adding CNY 80 for output gives a total of about CNY 1,130.
    • With caching: the first prefix write costs CNY 12.5. The next 99 calls hit the prefix, costing CNY 99. The added input costs CNY 0.5 per call, or CNY 50 in total. Output remains CNY 80. The total is about CNY 241.5.

    In this example, caching reduces total cost by about 79%. Actual savings depend on the hit rate and prefix length.

    6. How Should Safety and Factual Accuracy Be Controlled?

    LLMs can still generate incorrect facts. Vendor stress tests usually target confusing questions, while error rates in business scenarios need to be measured with your own data. In fields such as healthcare and law, authoritative knowledge bases and human review should be combined with model output rather than relying on the model alone.

    Some models filter high-risk data such as cyber offense and malicious code during training. This may reduce the model’s pass rate on extreme security tests, but it does not mean production environments are already safe. Enterprises still need red-team testing to check for prompt injection, unauthorized calls, and sensitive information leakage, and should use permission controls to limit the data and tools the model can access.

    Conclusion

    Million-token context allows models to process more complete material, but capability boundaries, cost structure, and security risks all need to be tested. Enterprises can use the testing methods in this article to build their own evaluation sets and use data to decide which model to adopt, when to use caching, and which operations must be confirmed by a human.