Evaluating LLM Java Coding Capabilities: Closed-Source Leads, Open-Source Catches Up

From Writing Functions to Modifying Projects: The Evolution of LLM Coding Evaluation

Evaluating the coding capabilities of large language models (LLMs) goes far beyond checking if they can write a simple bubble sort algorithm. Today’s mainstream evaluation frameworks have evolved into a clear three-tier structure:

1. Function-level algorithmic problems: Tests the model’s mastery of syntax and classic algorithms.
2. Repository-level tasks: Requires the model to locate issues and modify code within real-world projects containing hundreds or thousands of files.
3. Terminal-level operations: The model must independently write code, execute commands, read error logs, fix bugs, and re-verify, completing a full closed-loop process.

For Java developers, these three tiers perfectly map to daily work scenarios: writing utility classes, modifying business code, and troubleshooting production issues.

Among the widely recognized Java coding benchmarks, the function-level representative is a code generation benchmark (commonly referred to as LCB), which directly reflects a model’s ability to solve algorithmic problems and write functions. Repository-level representatives include long-range software engineering evaluations, terminal operation evaluations, and cross-repository multi-language evaluations. The latter covers over 550 real repositories and 33 programming languages, including Java. Terminal-level evaluations emphasize multi-step operations in a real terminal environment, simulating the entire process of delivering a feature from scratch.

A Clear Picture of 14 Models’ Java Capabilities: Closed-Source Takes the Lead

Let’s look at a comprehensive comparison table laying out the key scores of 14 mainstream LLMs. Note that blank spaces indicate the model did not publicly release scores for that specific benchmark, not a lack of capability.

Model Tier	Representative Model	Algorithmic Benchmark (LCB)	Repository-Level Task	Terminal Operation
Closed-Source Tier 1	Model A	92.8	41.2%	66.4
Closed-Source Tier 1	Model B	90.3	38.7%	64.1
Closed-Source Tier 1	Model C (Lightweight)	87.5	35.9%	61.8
Open-Source Tier 1	Model D	78.4	28.6%	49.2
Open-Source Tier 2	Model E	70.2	22.1%	40.5
Open-Source Tier 3	Model F	55.3	15.8%	31.7

Three key conclusions can be drawn from the table above:
First, the top three closed-source models form a structural lead across all three dimensions (algorithmic, repository-level, and terminal operations). In terminal operations specifically, they lead the strongest open-source model by over 15 percentage points.
Second, the strongest open-source model (Model D) is approaching entry-level closed-source performance on algorithmic problems, but a clear “engineering gap” remains in real-world repository tasks.
Third, model capability is not strictly positively correlated with parameter scale. A lightweight closed-source model actually outperforms larger open-source models on certain benchmarks.

The Snapshot Trap: Why Identical Models Show Drastically Different Scores Over Time

When comparing LLM coding capabilities, the most easily overlooked yet critical issue is the difference in benchmark snapshots. Taking LCB as an example, the benchmark regularly updates its question bank, leading to massive difficulty variations across different time snapshots. In an earlier snapshot, because the problems were relatively classic, multiple models achieved over 90% accuracy. Later, newly added problems involved more niche APIs and complex edge cases, causing the same models to generally drop to around 50%. Directly comparing scores across snapshots is like comparing total scores from two test papers of entirely different difficulties—it is meaningless.

Terminal operation benchmarks also suffer from version difficulty cliffs. Version 2.1 and Version 4.0 of a terminal evaluation benchmark may sound similar, but Version 4.0 introduced numerous scenarios requiring the handling of exception logs, concurrency race conditions, and network timeouts, making it far more difficult than 2.1. Therefore, if Model A scores 90 on Version 2.1 and Model B scores 66 on Version 4.0, you cannot directly conclude which one is stronger.

Practical Advice: When conducting model selection comparisons, ensure both parties use the exact same snapshot of the same benchmark. If vendors only publish scores from different snapshots, the safest approach is to obtain the test set and build your own evaluation suite, or at least ensure both models are evaluated within the same time frame and question bank.

Open-Source Catch-Up: Reaching the Ceiling in Algorithms, Facing a Gap in Engineering

The pace at which open-source models are catching up is truly impressive. Taking Model D as an example, it scored 78.4 on the algorithmic benchmark, only 9 points behind the third-place closed-source model at 87.5. A year ago, this gap was over 25 points. Furthermore, Model D supports an extremely long context window, making it quite practical for handling cross-file references in Java projects.

However, in scenarios closer to real-world development, open-source models still expose significant shortcomings. In repository-level tasks, Model D’s completion rate is only 28.6%, while the lowest among closed-source models is 35.9%. This means that out of 100 real GitHub issues (with problem descriptions), open-source models can independently fix fewer than 30. The gap stems from three main areas:

1. Understanding overall project architecture: Java projects often contain dozens of modules. The model needs to infer the best location for modifications from pom.xml dependencies, package structures, and design patterns.
2. Imitating existing code styles: Real projects often have strict Checkstyle rules and custom utility classes. Open-source models tend to write code “the way they are familiar with” rather than “the way the project requires.”
3. Multi-file collaborative modification: A single feature change often involves the Controller, Service, and Mapper layers. Open-source models struggle significantly to maintain logical consistency across these files.

Terminal Operations: The Litmus Test for “Real-World Development Capability”

Terminal operation benchmarks (such as long-range software engineering and terminal operation evaluations) are considered the litmus test because they strip away the ideal environment of “just writing code” and force the model to face the messy reality developers handle daily. In these tasks, the model must create files, compile, execute tests, read error logs, backtrack to modify code, and verify the fix. If any step fails, the entire task is considered a failure.

The top three closed-source models hold a structural lead in this dimension. When encountering compilation errors, they can more accurately determine whether it’s a syntax issue, a type inference issue, or a dependency conflict. When tests fail, they can distinguish between a business logic error and an issue with the test code itself. Open-source models are more prone to falling into a “blind trial-and-error” loop—modifying code multiple times without passing the same test, consuming numerous tokens without finding the root cause.

A deeper reason is that terminal-level tasks heavily test a model’s “memory management” capabilities. A long-range task often requires over 20 tool calls. The model must accurately remember the state after each operation and understand “how the variables introduced in the last modification affect the current operation.” The advantage of closed-source models here is linked to the richer real-world development logs seen in their training data.

Selection Advice and Benchmark Guide for Java Developers

Based on the above evaluation, here are three practical recommendations for Java developers:

First, select models by scenario, not by popularity. If you only need to write utility classes, generate unit tests, or explain code snippets, the experience gap between open-source and closed-source models is minimal. Prioritize open-source models to control costs. However, if you need a model to locate bugs in a large Spring Boot project or implement cross-module features, the top three closed-source models remain the safer choice.

Second, be wary of high scores on a single benchmark. A high score on an algorithmic benchmark only proves strong “paper coding” ability; it does not mean the model can handle your legacy systems. It is highly recommended to build a custom evaluation suite containing 10 to 20 real historical issues from your own team, and conduct regular regression tests.

Third, monitor the update cadence of the same model. In this evaluation, the performance of the lightweight closed-source model is noteworthy—using fewer parameters, it achieved algorithmic capabilities close to the top-tier models. Although a gap remains in terminal operations, it has already surpassed the open-source flagship models from half a year ago. This indicates that the primary bottleneck for model capabilities is shifting from “parameter count” to “training strategy.” In the future, lightweight models will see increasingly broad application in Java coding scenarios.

Conclusion: The “Three Laws” of Java Coding Capability

Reviewing the horizontal comparison of 14 models, we can summarize three fundamental facts about current LLM Java coding capabilities:

1. The overall gap between closed-source and open-source is narrowing, but the narrowing speed is fastest in the “algorithmic problems” dimension and slowest in “terminal operations.”
2. Benchmark scores must be interpreted in the context of version snapshots. Cross-snapshot comparisons will only yield misleading conclusions.
3. Java coding is not just about “language ability” but also “engineering ability.” A model’s depth of understanding regarding build tools, framework conventions, and design patterns is often a stronger determinant of the final outcome than code generation speed.

For developers, rather than chasing ranking changes with every evaluation, it is better to establish an evaluation methodology that suits your needs: clarify scenarios, pin down benchmarks, and maintain continuous observation. After all, true Java coding capability is not measured by benchmark scores, but by whether you can fix that production bug at three o’clock in the morning.