How to Evaluate Long-Context Capability: A Needle-in-a-Haystack Retrieval Test

Large language model context windows have expanded from a few thousand tokens to hundreds of thousands. A single request can now contain substantially more text, prompting teams to submit entire contracts, code repositories, or collections of technical documents directly to a model.

However, a larger context window also changes the evaluation problem. The fact that a model can accept a large amount of content does not mean it can accurately find and use the information that matters. This article explains how to assess that capability and provides a practical testing method that can be implemented directly.

1. Understand Tokens and Text Size First

A token is the basic unit a model processes. A tokenizer splits text into a sequence of tokens. The number of tokens in the same passage depends on the language, tokenizer, and content type.

English words are often close to one token, while a Chinese character may correspond to one or two tokens. Code, tables, and symbols are less predictable.

As a result, estimates such as “how many tokens are in several pages of a PDF” should be treated only as rough approximations. Layout, images, headers, and footers can all affect the result. When planning capacity, use the tokenizer for the target model to count tokens in representative, real-world content. Also reserve space for the system prompt, conversation history, and model output.

A practical estimation process is:

1. Extract representative documents and count their tokens with a tokenizer.
2. Calculate the average number of tokens per page or per thousand characters.
3. Subtract the space reserved for system prompts, conversation history, and output.
4. Estimate the amount of source material that can fit into a single request.

2. Fitting in the Context Window Does Not Mean the Model Can Use It Well

This is the central issue in evaluating long-context capability. Context size describes how much content a model can accept. Practical usability describes whether it can locate and correctly use the target information among a large amount of irrelevant material.

Three common sources of bias are:

• Position bias: Information placed in the middle of a document is often less likely to be retrieved than information at the beginning or end.
• Interference bias: As more similar passages are added, the model is more likely to confuse entities, dates, and numerical values.
• Degradation with scale: As the input grows longer, answers to the same question become less stable, and repeated calls with the same input may produce more varied results.

This can be compared to a library. A larger shelf does not guarantee that every book can be found quickly. Shelf capacity determines how many books can be stored; the librarian’s indexing ability determines which books readers can locate.

3. How to Design a Needle-in-a-Haystack Test

The needle-in-a-haystack approach places a single relevant “needle”—a clear factual statement—inside a large amount of irrelevant text. The model is then asked a question to determine whether it can accurately retrieve that information.

The test has three key variables:

• Context length: Increase the input from a short text to the target size, such as 1K, 8K, 32K, and 128K tokens.
• Needle depth: Define the needle’s relative position in the document. A depth of 0% means the beginning, while 100% means the end.
• Needle content: Use a fictional fact that is unrelated to the background material and has a unique answer, making automated scoring easier. Fictional content also prevents the model from relying on knowledge memorized during training.

Each length-and-depth combination should be tested multiple times. Calculate the accuracy and plot the results as a two-dimensional heat map. This makes performance boundaries easier to identify.

Construction Steps

1. Prepare background material that is close to the target business domain but unrelated to the needle.
2. Design the needle and question. For example, the needle could state, “The internal code name of the project is XX-7,” and the question could ask, “What is the internal code name of the project?”
3. Truncate the background material to the target length and insert the needle at the specified depth.
4. Submit the question and collect the model’s response.
5. Use rule-based matching or independent scoring logic to determine whether the response contains the correct answer.
6. Aggregate the results and analyze failed samples individually.

The following is a simplified example. The call_model function should be implemented by the user to call the target model. This example controls length by sentence count; a formal evaluation should truncate content by token count.

import random
def build_context(background_sentences, needle, depth_ratio):
    """Insert the needle at a proportional depth from 0 to 1."""
    position = int(len(background_sentences) * depth_ratio)
    sentences = background_sentences[:]
    sentences.insert(position, needle)
    return "\n".join(sentences)
def run_trial(call_model, background_sentences, needle, question, answer, depth_ratio):
    context = build_context(background_sentences, needle, depth_ratio)
    prompt = f"Read the following material and answer the question.\n\n{context}\n\nQuestion: {question}"
    reply = call_model(prompt)
    return answer in reply
def run_grid(call_model, background_sentences, needle, question, answer,
             lengths, depths, repeats=3):
    results = {}
    for length in lengths:
        subset = random.sample(
            background_sentences,
            min(length, len(background_sentences))
        )
        for depth in depths:
            hits = sum(
                run_trial(
                    call_model,
                    subset,
                    needle,
                    question,
                    answer,
                    depth
                )
                for _ in range(repeats)
            )
            results[(length, depth)] = hits / repeats
    return results

4. How to Interpret the Results

When interpreting the results, focus on the following areas:

• Capability boundary: The regions where failures are concentrated on the heat map represent areas outside the model’s reliable working range. Business inputs should stay within this range or be split first.
• Position-specific failures: If failures occur only at certain depths, consider moving important information closer to the beginning or end, or repeating key constraints in the prompt.
• Result variability: If repeated tests for the same combination produce substantially different results, the sample size may be too small. Increase the number of repetitions.
• Test coverage: This test evaluates single-point factual retrieval only. It does not represent reasoning, summarization, or cross-passage comparison capabilities. Those capabilities require separate test sets.

Performance can vary significantly across models and model versions. Decisions should be based on results from the organization’s own business data rather than directly applying figures reported by others.

5. Controlling Testing Costs

The main cost of large-scale long-context testing comes from input tokens. The number of calls can be estimated as follows:

Number of calls = number of length levels × number of depth points × number of repetitions

Total cost can be approximated as:

Total cost ≈ average input tokens per call × price per token × number of calls

Because the longest length levels contain far more tokens per call than the shortest ones, most costs are concentrated in the longest tests. Common ways to control costs include:

1. Start with short texts and a coarse grid, such as testing every 25% of the document, to identify the approximate boundary. Then run more detailed tests near that boundary.
2. Reuse the same background prefix across tests if the service supports an appropriate caching mechanism.
3. Run large-scale tests in a test environment using quotas or a sandbox instead of production traffic.
4. Limit the number of length levels and avoid adding unlimited sampling points merely to make the curve appear smoother.

6. Engineering Recommendations

Long-context processing can complement retrieval, but it should not be treated as a replacement for retrieval. Common implementation patterns include:

1. Retrieval augmentation: Use keyword or vector search to select relevant passages first, then provide them to the model to reduce irrelevant content and interference.
2. Place important information strategically: Put the question and constraints at the beginning or end of the prompt, and place important source material close to the question.
3. Require traceable output: Ask the model to identify the paragraph containing the answer or quote the original sentence so that the result can be verified manually.
4. Summarize in stages: For extremely long material, summarize it in segments first, then process the summaries in a second step while preserving paragraph indexes.
5. Monitor regressions: After upgrading a model or version, rerun the same test set and compare the results.

In an enterprise environment, original documents can be stored in object storage, indexes can be built with a general-purpose retrieval service, and batch evaluation tasks can run through a message queue or batch-processing queue. Selection should also account for data security and compliance requirements, including whether documents are permitted to enter an external model service.

7. Implementation Checklist

• Use the target model’s tokenizer to measure token counts for real-world content.
• Use fictional content for the needle and verify that it does not overlap with likely training data.