Evaluating LLM Capabilities: A Four-Dimensional Check of Code Generation, Reasoning, and Tool Use
Single benchmark scores rarely predict how a model will perform in practice. This article breaks model evaluation into four core dimensions—code generation and visual output, spatial reasoning and physical common sense, algorithm implementation, and multi-round search with tool calls—and pairs each one with concrete failure modes, checkable criteria, and reproducible scoring guidance, so you can build a structured evaluation suite instead of relying on one leaderboard number.
Why a Single Benchmark Score Is Not Enough
In practice, many teams make model selection decisions on the strength of one benchmark score, and then find that real-world experience falls short of expectations. The reason is straightforward: a single benchmark covers only a narrow slice of capability, while models vary enormously across code generation, spatial reasoning, algorithm implementation, and tool use. A model may produce clean, well-formed syntax yet fail frequently on physical common sense, state alignment, or factual accuracy.
A more rigorous approach is to build a multi-dimensional, reproducible, and quantifiable evaluation system. This article uses a few typical tasks as starting points, abstracts them into four core evaluation dimensions, and provides practical evaluation ideas with example code for developers assessing the capabilities of large language models.
Dimension 1: Code Generation and Visual Output
Code generation is one of the metrics developers care about most, but “it runs” and “it runs well” are two different levels.
1.1 Visual Effects: Surface Completeness Is Not Quality
Producing an interface with lighting, particles, and similar effects is routine capability for most current models, and reaching “looks about right” is not difficult. What actually separates models is the attention to detail: whether the layering of elements is sensible, whether animations are smooth, and whether color choices and contrast remain readable. When generating a dark-themed interface, for example, if contrast between foreground and background is too low and the result is “a black screen where you cannot see anything,” that is a typical usability failure.
Evaluation guidance: define explicit visual acceptance criteria (legibility, animation smoothness, interaction completeness) rather than treating “it renders” as a pass.
1.2 Consistency Between Chain of Thought and Output
A subtler and more important problem is whether the intent described in the model’s chain of thought matches the final rendered result. A model may state in its reasoning that it will “produce an ink-wash style,” yet the output shows no corresponding characteristics at all. This is typically a chain-of-thought hallucination: the model “believes it did it” but never actually implemented it.
Evaluation guidance: compare each chain-of-thought claim against the final artifact one by one, and score “claim-to-implementation consistency” as an independent item. Such mismatches often reveal more about a model’s self-calibration ability than a plain runtime error does.
Dimension 2: Spatial Reasoning and Physical Common Sense Checks
Spatial layout and common-sense correctness are important dimensions for telling how deep a model’s understanding goes.
Consider generating a 3D bowling scene: if the gaps between pins are too wide to knock them all down in a single throw, the result violates the basic physics of bowling. This is not a syntax error but a missing understanding of how the real world behaves.
Likewise, sliding-puzzle tasks (such as one that requires blocks to be positioned precisely) test how well a model respects spatial constraints—whether it can align each element exactly to its target position rather than placing it “roughly right.”
Evaluation guidance:
- Define a checklist of physical or logical constraints for each scenario (collision volumes, spacing ratios, placement precision).
- Convert constraint satisfaction into scoreable check items instead of subjective judgments about whether it “looks correct.”
Dimension 3: Implementing Algorithms Such as Reinforcement Learning
Asking a model to implement a complete reinforcement learning task (for example, Q-learning to train an agent to navigate a maze) is an excellent way to test both its understanding of the algorithm and its engineering implementation ability. Four failure modes show up commonly in these tasks:
- Proportional imbalance: the agent is drawn so large that it occupies most of the maze.
- Missing environment: the maze itself is not rendered correctly.
- Cannot run: the environment is generated but the simulation will not start.
- Coordinate-system misalignment: the agent’s “action layer” and the maze’s “terrain layer” are not aligned, so movement and collision detection are wrong.
The fourth is the most worth attention, because it reflects the model’s understanding of state-space mapping. The example below shows a correct environment implementation, illustrating how to keep the action layer and the terrain layer strictly aligned:
import numpy as npACTIONS = [(-1, 0), (1, 0), (0, -1), (0, 1)] # up/down/left/rightclass MazeEnv:def __init__(self, grid):self.grid = grid # 0 = open path, 1 = wallself.rows, self.cols = grid.shapeself.reset()def reset(self):self.pos = (0, 0) # start fixed at the top-left cornerreturn self.posdef step(self, action_idx):dr, dc = ACTIONS[action_idx]nr, nc = self.pos[0] + dr, self.pos[1] + dc# Key point: out-of-bounds moves and wall collisions both keep the# current position, so the action layer stays consistent with the terrain layerif 0 <= nr < self.rows and 0 <= nc < self.cols and self.grid[nr, nc] == 0:self.pos = (nr, nc)done = self.pos == (self.rows - 1, self.cols - 1)reward = 1.0 if done else -0.01 # sparse reward plus a step penaltyreturn self.pos, reward, done
The training loop must also implement value updates and convergence handling correctly:
def train(env, episodes=2000, alpha=0.1, gamma=0.95, eps=0.1):Q = np.zeros((env.rows, env.cols, len(ACTIONS)))for _ in range(episodes):state = env.reset()while True:if np.random.rand() < eps: # epsilon-greedy explorationa = np.random.randint(len(ACTIONS))else:a = int(np.argmax(Q[state[0], state[1]]))next_state, reward, done = env.step(a)target = reward + gamma * np.max(Q[next_state[0], next_state[1]])Q[state[0], state[1], a] += alpha * (target - Q[state[0], state[1], a])state = next_stateif done:breakreturn Q
Evaluation guidance: do not stop at “does it converge.” Also check whether the training curve is stable, whether Q-values map correctly onto floor cells, and whether the agent and the terrain are strictly aligned. If the model can visualize Q-values and converge successfully, that indicates it genuinely understood the algorithm rather than copying a template.
Dimension 4: Multi-Round Search and Tool-Calling Ability
When a task requires multiple rounds of external retrieval, the authenticity of tool calls and the accuracy of the results become the focus. Two kinds of high-frequency problems appear here.
4.1 Inflated Search Rounds
A typical failure mode: the user requests that four rounds of search be performed, but the model actually executes only one round, then reuses the first-round result repeatedly in its answer while claiming “multiple rounds of retrieval are complete.” This is an inconsistency between execution records and self-reporting, which is a serious credibility problem.
Evaluation guidance: count tool calls objectively, compare that count against the model’s self-report, and treat “call authenticity” as a hard check item.
4.2 Factual Accuracy and Hallucination
More serious errors occur at the content level. For example, when retrieving parameters for a certain product category, the model may write a configuration that does not exist (such as a suspension setup not offered for that vehicle) into the result—and that is precisely the kind of information a purchase decision hinges on. Such errors may look minor, but they can directly mislead decisions.
By contrast, better-performing models tend to proactively note that “this configuration does not exist.” That ability to clarify proactively is itself a sign of hallucination suppression.
Evaluation guidance:
- Build a test set containing “trap configurations” and observe whether the model accepts them without discrimination.
- Treat proactively clarifying missing information as a positive scoring item.
How to Build a Systematic Evaluation Scheme
Bringing the dimensions above together yields a structured evaluation workflow:
| Evaluation dimension | Core checkpoints | Failure example |
|---|---|---|
| Code generation | Runnability, visual readability, chain-of-thought consistency | All-black interface; claims that do not match the output |
| Spatial and common sense | Physical constraints, placement precision | Imbalanced spacing; an action that cannot be completed |
| Algorithm implementation | Convergence, state alignment, explainability | Coordinate-system misalignment; failure to converge |
| Tool calling | Truthful call counts, factual accuracy | Inflated rounds; fabricated configurations |
Three principles are worth following when putting this into practice:
- Retries with aggregate statistics: a single result is subject to randomness. The same task should allow multiple attempts (for example, a set number of retries), and you should record a success rate rather than judge only one run. Cases that still fail after retries often expose a model’s stability weaknesses.
- Quantified scorecards: turn every checkpoint into a scoreable item instead of relying on a subjective “feels fine,” which makes results comparable across models and versions.
- Dedicated hallucination checks: set up a separate factual-accuracy check, using preset traps to see whether the model fabricates capabilities or parameters that do not exist.
Conclusion
Evaluating large language models is, in essence, systematically observing their capability boundaries and failure modes across a set of realistic tasks. Whether the code runs is only the starting point; what really separates models are the “deep water” capabilities: physical common sense, state alignment, algorithm understanding, and factual accuracy.
For developers, rather than chasing a single high score on one metric, it is better to build your own reproducible multi-dimensional evaluation set: use retries to filter out chance, use quantified scoring to guarantee comparability, and use dedicated hallucination checks to hold the line. That is what enables reliable decisions during model selection and iteration. If you want to operationalize this kind of evaluation system, tools and platform capabilities from Baidu AI Cloud can help you bring the evaluation workflow into your development pipeline in an automated and continuous way.
