LLM Internal State Interpretability: Reading the Hidden Reasoning of Large Language Models

1. Behind the Answer: An Invisible Reasoning Chain

When a large language model is asked, “How many legs does an animal that builds webs have?”, it may answer “8” almost immediately. This response can look like a simple knowledge lookup, but the model usually performs at least two internal steps:

1. It infers that “an animal that builds webs” refers to a spider.
2. It uses “spider” as an intermediate concept to retrieve the common knowledge that spiders have eight legs.

The key point is that the intermediate concept “spider” never appears in the final output. Yet it must have been activated and used somewhere inside the model; otherwise, the model would not have reached the correct answer. This kind of internal state—activated and used during computation, but not explicitly shown in the output—is one of the central objects studied in large language model interpretability research. It is often informally described as the model’s “hidden thoughts.”

Understanding these hidden thoughts matters because the model’s output is only the final stage of a much longer computation chain. The information that determines whether the model is correct, and why it may fail, is often embedded in hidden-layer vector representations. Teams that can interpret these intermediate states are better positioned to diagnose hallucinations, bias, and reasoning errors, and to build more trustworthy model evaluation systems for critical business scenarios.

2. Why Internal Representations and Explicit Outputs Can Diverge

From an engineering perspective, a large language model does not simply “think through an answer and then report it faithfully.” Instead, the model passes input through multiple layers of nonlinear transformations and eventually projects the result into a probability distribution over the vocabulary. The final token is then sampled from that distribution.

The semantic information carried by intermediate layers is often richer and more abstract than the final output token. However, only signals directly related to generating the “next token” are explicitly expressed. This separation can be understood at two levels:

• Computational depth: Complex reasoning is often multi-hop. In chains such as “builds webs → spider → eight legs,” the more intermediate steps there are, the more intermediate concepts may remain only in hidden states and cannot all be packed into the final sentence.
• Alignment and training objectives: Models are often trained to provide concise and polite answers. As a result, many internal trade-offs, hesitations, and rejected candidate answers are compressed away. This is somewhat similar to a person who considers several options internally but only says one polished sentence aloud. Internal states and external expression can therefore be clearly inconsistent.

Because of this, observing only the output is not enough to fully reconstruct the model’s decision process. To evaluate what the model is actually doing, it is necessary to directly inspect its internal representations.

3. Three Interpretability Approaches: Capabilities, Strengths, and Limits

Several technical routes have emerged around the goal of reading internal states. They mainly differ in observation perspective—whether the analyst looks at output distributions, individual features, or concept subspaces—and in granularity.

1. Output-layer view: Logit Lens

Logit Lens takes a direct approach: it projects the hidden state produced by each layer into the output vocabulary space and observes which tokens that layer is “inclined” to produce.

• Strengths: It is simple to implement and usually adds little computational overhead. It can quickly help locate the layer at which the model begins to form an answer tendency and is commonly used to track changes in confidence across layers.
• Limitations: It can only show the distribution over the “next token.” It cannot reveal intermediate concepts that may be used later but are not currently output. In this view, reasoning bridges such as “spider” are often invisible.

2. Feature view: Sparse Autoencoder Analysis

Sparse autoencoders attempt to decompose high-dimensional, highly entangled hidden states into a large set of sparser and more interpretable features. Each feature may correspond to a nameable semantic unit, such as an entity, a grammatical role, or an emotional tendency.

• Strengths: The granularity is much finer than that of output-layer probes. Entangled representations can be broken down into thousands of relatively independent features, making manual interpretation and statistical analysis easier.
• Limitations: The number of features can be very large, and many may be noisy. It is difficult to directly distinguish which features are truly serving the current reasoning task and which are merely byproducts of the model’s automatic processing. In other words, sparse autoencoders are good at discovering features, but not necessarily at determining their causal role.

3. Concept-subspace view: Jacobian-based Localization

The third class of methods focuses on a more precise question: how sensitive is the final output to changes in a hidden state? By computing the Jacobian of the hidden state with respect to the output, one can estimate the “leverage” of each internal dimension on the output. This makes it possible to locate subspaces corresponding to concepts that are strongly activated, causally influential, and yet absent from the final output.

• Strengths: This approach does not merely look at activation strength; it looks at causal influence. It is therefore closer to answering whether an intermediate concept truly participated in the reasoning process. The “spider” example above is well suited to this kind of method.
• Limitations: Jacobian-based computation can be expensive. Layer-by-layer analysis on large models still requires engineering optimization. In addition, mapping a discovered subspace to a human-nameable concept often requires extra mapping and validation.

4. Comparison and Practical Selection

Dimension	Output-tendency probe	Sparse autoencoder analysis	Jacobian-based concept localization
Observation target	Output token distribution	Individual semantic features	Concept subspaces that influence the output
Granularity	Coarse: token level	Fine: feature level	Medium: concept level
Causal strength	Weak	Medium	Strong
Computational cost	Low	Medium	High
Typical use	Tracking answer formation	Large-scale feature discovery	Precisely locating reasoning bridges

In practice, these methods are often complementary rather than mutually exclusive. A common workflow is to use output probes to quickly locate key layers, then use sparse feature analysis to narrow down candidate concepts, and finally use Jacobian-based methods to confirm which concepts actually participate in the reasoning process. This can significantly improve both the efficiency and reliability of interpretability analysis.

5. General Interpretability Workflow

The following code provides a generic illustrative skeleton for an interpretability analysis workflow. It is intended only to show the general process; specific interfaces should follow the framework being used.

import torch
def collect_hidden_states(model, input_ids):
    """Collect hidden states layer by layer for later analysis."""
    states = []
    with torch.no_grad():
        hidden = model.get_input_embeddings()(input_ids)
        for layer in model.layers:
            hidden = layer(hidden)
            states.append(hidden.detach().clone())
    return states
def logit_lens(head, hidden_state):
    """Project a layer's hidden state into the vocabulary to inspect its output tendency."""
    logits = head(hidden_state[:, -1, :])      # Use the last position
    probs = torch.softmax(logits, dim=-1)
    topk = torch.topk(probs, k=5)
    return topk.indices, topk.values           # Candidate tokens most favored by this layer
def jacobian_influence(hidden_state, output, concept_dim):
    """Roughly estimate the sensitivity of the output to a hidden dimension (concept influence)."""
    h = hidden_state[:, -1, concept_dim].clone().requires_grad_(True)
    out = output.sum()
    out.backward()
    return h.grad.abs().mean().item()          # Average leverage of this dimension on the output

A typical analysis workflow consists of four steps:

1. Prepare probe data: Construct input samples that can trigger the target reasoning chain, such as multi-hop commonsense questions.
2. Collect hidden states layer by layer: Preserve intermediate representations as the objects of analysis.
3. Compare multiple views: Compute output tendencies, sparse feature activations, and concept influence scores together.
4. Validate causality: Run intervention experiments on suspected “reasoning bridge” concepts by activating or suppressing them, and observe whether the output changes accordingly. This helps confirm the actual role of the concept.

6. Engineering Value and Cautions

Incorporating interpretability into the lifecycle of large language model applications can provide several practical benefits:

• Fault diagnosis: When a model hallucinates or gives an irrelevant answer, interpretability tools can help identify which layer and which concept contributed to the deviation, rather than simply retrying and hoping for a better result.
• Reliability evaluation: In high-risk domains, such as content moderation or financial risk-control assistance, interpretability analysis can help determine whether a model’s conclusion is supported by substantive internal reasoning rather than merely appearing plausible.
• Guidance for continuous optimization: Interpretability metrics can provide observable signals for prompt engineering, fine-tuning data selection, and model selection.

Several cautions are also important:

• Avoid over-attribution. Visualized “concepts” are approximations from a human perspective. They do not necessarily correspond to a discrete symbol that truly exists inside the model, so interpretations should remain cautious.
• Distinguish correlation from causation. Strong activation does not necessarily mean the concept causally affects the output. Intervention experiments should be used to support causal claims whenever possible.
• Balance cost and benefit. Jacobian-based methods can be computationally expensive. It is usually better to apply them selectively, layer by layer, or on sampled data rather than running full-scale, all-layer analysis.

For practical deployment, teams can build an interpretability pipeline using generic cloud capabilities: object storage can archive inference samples and intermediate states, container platforms can schedule batch analysis jobs, and monitoring, alerting, and logging services can help track abnormal changes in model behavior. These are general-purpose capabilities and can be combined flexibly according to the existing technology stack.

7. Limitations and Future Directions

Interpretability techniques for large language models are still evolving rapidly. Output probes can be too coarse, sparse features can contain considerable noise, and concept localization can be expensive. No single method can fully reconstruct the model’s “hidden thoughts” by itself.

Three directions are likely to be important:

1. Fuse multiple views: Combine output probes, feature analysis, and Jacobian-based methods into a closed loop of “locate—interpret—validate.”
2. Reduce the cost of high-cost methods: Make Jacobian-based and other expensive techniques more practical for larger models.
3. Move interpretability earlier in the lifecycle: Shift interpretability from post-hoc analysis into training and alignment stages, improving behavioral transparency at the source.

Conclusion

The answer produced by a large language model is only the tip of the iceberg. The reasoning that truly determines its behavior often lies in internal states that are never explicitly output. Understanding and using these “hidden thoughts” is a key step toward improving model trustworthiness, diagnosing errors, and supporting engineering decisions. For developers, the important shift is not only to observe final outputs, but also to build the ability to read, validate, and intervene in internal representations. This is the dividing line between merely using a model and truly understanding it.