Limits of Large Language Models in Reverse Engineering

In software engineering, reverse engineering is rarely a matter of reading a binary once and understanding it completely. It is usually a gradual reconstruction of control flow, data flow, and runtime conventions under limited symbols, limited comments, and few known-correct samples. Large language models are now widely used to explain assembly snippets, infer function intent, and organize call relationships. They can lower the barrier to entry, but treating a model as a fully automatic decompiler quickly runs into correctness limits. Those limits matter: models are most useful for explaining a single instruction, filling in terminology, and producing a list of hypotheses, while the final conclusion stays with human cross-checking.

At a fundamental level, a large language model is next-token prediction conditioned on context. Its “understanding” of assembly comes from patterns in training data: common instruction mnemonics, typical function prologues, and familiar loop and memory-copy templates. High-quality, human-reviewed reverse-engineering corpora are scarce. Even experienced analysts often misread local details when facing optimized control flow, tail calls, indirect jumps, and runtime selection. A model cannot invent a more reliable answer key than the people who produced the training material, so on complex fragments it often looks locally plausible while the overall meaning drifts.

A typical case is an optimized memory-copy implementation. Compilers and standard libraries often generate several paths for different lengths, alignments, and processor features, then choose the real entry through a runtime mechanism. If an analyst pastes a long assembly block with many jumps into a model, different models may tell incompatible stories: one may call it ordinary loop unrolling, another may invent branch conditions that are not there, and another may miss the layer that probes capabilities and then jumps to a specialized implementation. A more reliable approach is to break the problem down. First decide whether the code is library dispatch or business logic. Then check comparisons, condition codes, and jump targets one by one. Only after that should the model restate, in natural language, paths that have already been verified. In practice, instruction-level explanations are usually highly usable. Accuracy drops clearly once the task spans multiple basic blocks, jump tables, or lazy binding.

A common misconception is that open source automatically means the model already understands the binary. Even when a high-level memory-copy implementation is public, what the analyst actually gives the model is often a slice of assembly from a specific build: a particular optimization level, target architecture, and link options, bound to runtime symbol resolution. If the model is not told that the fragment is a dispatcher rather than the copy loop itself, it tends to force an instruction-level reading and produce more wrong claims. A better prompt states constraints: target architecture, calling convention, whether indirect calls are present, how a jump-table base is computed, and facts already confirmed—for example, which class of implementation is actually loaded on the board. Writing known facts into the prompt is more effective than hoping the model will guess the mechanism’s name.

Another practical case is turning engine calls that live inside a scripting environment into a standalone binary. Many professional applications expose script bindings: the interpreter process loads a native dynamic library and passes commands through in-process calls rather than a network protocol. To leave the script runtime and wrap the engine as an executable, you need to understand how the install root and library paths are resolved, how the dynamic library is loaded, how exported symbols map onto the command channel, and how session state is kept in the same address space. The work can be split into three steps. First, read only the script-side configuration: infer the install root from the current file location, set system directory variables, and open the engine library with dynamic loading. Second, narrow the symbol set. Do not decompile tens of thousands of functions at once; that quickly exhausts memory and can crash the analysis session. Start from function names that appear in the script bindings and build a candidate set of imported symbols, possible implementations, and call sites. Third, decompile only those candidates and use cross-references to check argument layout and lifetime.

For large binaries, memory and interaction strategy matter as much as the model. A full decompile of tens of thousands of functions can stall under limited memory. A better allocation of effort is to search symbols and strings first and lock onto clusters related to command execution, session setup, and result return; build control-flow graphs for functions inside those clusters instead of deeply decompiling the whole library; and use the model on short excerpts you have already cut out, not on “read the entire library and summarize.” When the model’s explanations contradict each other, drop back to the instruction level: how flags affect later branches, how an indirect-jump index is computed from length or capability bits, and how relocations and the procedure linkage table turn a symbol into a real address. After a person organizes those fragments, asking the model to turn them into a structured write-up is usually more efficient than one long prompt.

For analysts, model capability falls into three layers. The first is vocabulary and patterns: mnemonics, addressing modes, common prologues and epilogues, and calling-convention differences. This layer has the largest payoff and the smallest risk. The second is local procedure structure: a basic block of roughly ten to several dozen instructions. The model may suggest “this looks like a copy loop” or “this looks like length-based dispatch,” but the claim must be checked against actual jump targets. The third is system semantics: runtime function selection, dynamic loading, in-process embedding, and object lifetime across language boundaries. Here the model often fills gaps with a familiar story from training data, and those stories need to be falsified with external documentation, symbols, and dynamic tracing. Using the layers separately avoids treating fluent prose as a verified conclusion.

A reusable workflow looks like this. In preparation, fix the target architecture and calling convention, record build options and load paths, and list facts already confirmed. When asking questions, cover one basic block or one path at a time, require the model to mark uncertainty, and forbid it from inventing symbol names that do not appear. In verification, check every branch and memory access in a disassembler; if needed, break in a debugger at the dispatch point and observe the real entry. In summarization, a person writes the paths as a table of conditions, targets, and side effects, then the model only improves readability. If the goal is to package a script-embedded engine as a standalone binary, also check whether library dependencies can be loaded statically or explicitly, whether global state is thread-safe, how error codes are mapped, and whether resource release matches the original script binding.

Compliance and engineering risk still apply. Reverse engineering a third-party closed-source engine should stay within lawful authorization and the minimum needed for compatibility research, and should not spread unpublished interface details. Keep the analysis environment separate from production, cap memory, and avoid letting a full decompile take down a workstation. Do not paste strings related to authorization, keys, or user data into prompts. If model output mentions an undocumented interface, treat it only as a hypothesis and confirm it with observed behavior.

Overall, large language models are a real help in reverse engineering, and that help has clear limits. They can turn instruction-level noise into hypotheses worth discussing, and they can speed documentation once a mechanism has been stated explicitly. They cannot replace human checks of jump tables, runtime dispatch, and dynamic linking, and they cannot guarantee global correctness on complex control flow where training data is thin. Sustainable use means cutting tasks small, writing known facts into the context, and binding conclusions to observable behavior. Teams that need this at scale can also record session crashes and memory peaks with logging and monitoring alerts, and archive extracted function fragments in object storage, so human–model reverse engineering becomes a repeatable engineering process rather than a one-off experiment.