How to Evaluate an LLM's Engineering Capability: Lessons from a Lock Placement Mistake

Discussions about which large language model is the “best” often surface in developer communities. But for engineers working on enterprise projects, the more pressing question is rarely about rankings. It’s about which model can be used reliably under real project constraints—where it runs, how it behaves when instructions are incomplete, and whether its output aligns with the overall system design.

This article walks through a concrete failure case to illustrate what separates models that merely “pass tests” from those that respect architectural intent.

Start with the Constraints: Who Is the Model Serving?

Many developers choose a model by looking at leaderboard scores. But in enterprise settings, especially government or compliance-heavy projects, the first constraint is not capability ranking—it is the hard boundary of the business scenario.

For example, projects with strict data compliance requirements often mandate that data never leaves the controlled environment. Model services must run inside the infrastructure boundary, and critical systems may require fully private deployment. Under these conditions, a model with excellent public benchmarks but no compliant deployment option is simply not a candidate.

This context changes how you should evaluate a model. Before asking “which model is stronger,” ask which models are allowed in your environment, then assess capability within that set.

Parameter Scale Is Really About Intent Understanding

A common assumption is that two models with similar benchmark scores will behave similarly in practice. That assumption often fails—specifically in the layer of intent interpretation.

Smaller parameter models tend to behave as “instruction executors”:

• They faithfully follow the literal text of your prompt, but ignore what is not explicitly stated.
• Prompts must be exhaustive: boundary conditions, exception branches, naming conventions, lock granularity—nothing can be left out.
• When ambiguity exists, these models fill the gap with their own “default assumptions,” which frequently diverge from what you actually designed.

Larger parameter models, by contrast, tend to act as “intent inferrers”:

• They synthesize context—table structures, descriptions of business relationships, prior conversation—to infer what you really want.
• They handle ambiguous instructions with better disambiguation.
• Where you haven’t specified details, they lean toward engineering conventions and consistency with existing code style.

Why doesn’t this show up in benchmarks? Because most evaluation sets are structured as “clear input, definite answer” problem-solving tasks. They measure knowledge and reasoning. But in real development, much of the value lies in reconstructing design intent from fuzzy requirements—a dimension that standard benchmarks rarely capture.

That said, this deeper understanding comes at a cost. Larger models require more inference resources, and that economic trade-off is a legitimate part of engineering selection.

A Concrete Case: The Exclusive Lock Was Placed on the Wrong Entity

Consider a real project involving a data and archive management module. The domain model is simple: archives are attached to data records. Each data record can have multiple archives, and each archive belongs to a folder.

The intended design was:

1. Acquire an exclusive lock on the data record’s primary key (id).
2. Perform archive operations within the same transaction.

Two benefits follow from this design:

• A single lock point covers all operations on that data record, simplifying lock management.
• It prevents a specific race condition: while an archive operation is in progress, another user might delete the entire data record. That would leave the archive pointing to a non-existent record—an orphan.

The intended implementation looks like this:

BEGIN;
-- Lock the data primary key: all operations on this data record are mutually exclusive
SELECT * FROM biz_data WHERE id = #{dataId} FOR UPDATE;
-- Write the archive within the same transaction; no second lock needed
INSERT INTO biz_archive (data_id, folder_id, file_url)
VALUES (#{dataId}, #{folderId}, #{fileUrl});
COMMIT;

The smaller model produced a different implementation. Because the prompt did not explicitly say “lock the data record’s primary key,” it defaulted to locking the archive folder instead:

BEGIN;
-- Locks the archive folder; lock target does not match design intent
SELECT * FROM archive_folder WHERE folder_id = #{folderId} FOR UPDATE;
INSERT INTO biz_archive (data_id, folder_id, file_url)
VALUES (#{dataId}, #{folderId}, #{fileUrl});
COMMIT;

Does this implementation work? Yes. Does it pass tests? It can. In single-threaded test cases, the two versions behave identically. The demo looks flawless.

“Works in a Test” vs. “Matches the Design”: A Wide Gap

The difference only emerges under concurrency:

Concurrent Scenario	Lock on Data Primary Key (Design Intent)	Lock on Folder ID (Model Implementation)
User A operates on an archive while User B deletes the same data record	B is blocked; archive operation completes safely	B is not blocked; the archive may become an orphan pointing to a deleted record
Archive operations on different folders of the same data record	Naturally serialized; no conflict	Runs in parallel; operations may interfere with each other
Number of lock points	One; logic concentrated	Grows with the number of folders; lock management becomes more complex

The more serious issue is debugging cost. Tests are green, demos look fine, and the problem only appears under a specific concurrency interleaving. Worse, the failure mode is dirty data, not an explicit error. This kind of defect is nearly invisible during acceptance testing. When data anomalies surface in production, tracing back the root cause becomes disproportionately expensive.

This case highlights a frequently overlooked fact: software correctness is not just “functionally correct.” It also includes consistency with the overall design. Where the lock is placed, and at what granularity, is an architectural decision. A model cannot arbitrarily choose a “nearby” lock target. The failure mode of smaller models is to select a locally plausible default where information is missing. Engineering design requires global consistency. Larger models tend to be more reliable in such cases because they can combine context like “archives belong to data records” and infer that the lock anchor should be the lifecycle subject of the data—not the organizational structure of the archive. That cross-context inference depth is a core benefit of parameter scale.

Wish-Driven Coding vs. Contract-Based Development: They Are Compatible

A popular practice is sometimes called “wish-driven coding”: treat the request as a wish, hand it to the model, and quickly get a runnable result. This approach is not in conflict with the argument in this article. They are two ends of a spectrum:

• Wish-driven coding aims at “getting something built.” It suits prototypes, personal tools, and one-off scripts.
• Formal enterprise projects aim at “building it according to the design.” Models must follow explicit and implicit contracts: database constraints, transaction boundaries, lock protocols, interface agreements, coding conventions—all aligned with the engineer’s design path.

A development team can use a lightweight model for rapid prototyping, then switch to a larger model for engineering phases, while tightening prompt constraints and review processes. Different stages require different tools and different discipline.

Practical Recommendations: Model Allocation and Prompt Engineering

Based on the analysis above, here are five actionable recommendations:

1. Assign models by task type. Design-sensitive tasks with complex context or ambiguous boundaries—architecture discussions, core business logic, concurrency control—should go to larger models. Tasks with clear input/output patterns—template code, format conversion, comments and documentation—are better handled by smaller models for cost efficiency.

2. Write boundary conditions as explicit checklists. Instead of complaining later that the model missed something, remove ambiguity upfront in the prompt:

   [Locking Requirements]
   - Lock target: biz_data.id (data primary key), explicitly do NOT lock archive_folder
   - Lock type: exclusive lock (SELECT ... FOR UPDATE)
   - Lock scope: covers the entire archive operation transaction
   - Design purpose: prevent concurrent deletion of the data record during operations

3. Align on design before generating code. Ask the model to restate its understanding of the design intent first: “What entity do you plan to lock, and why?” Confirm this before any code is written. Most misalignment can be caught at this stage.

4. Concurrency logic requires manual review. Code involving transactions, locks, or timing sequences should never be accepted based on tests alone. Manually reason through concurrent execution paths, and verify that lock granularity and lock placement match the architectural decision.

5. Verify deployment mode for compliance-sensitive scenarios. For teams with strict data-residency requirements, prioritize models where the service deployment can be controlled. When evaluating cloud providers, review whether the model service supports private deployment, data isolation, and audit capabilities that match your compliance requirements.

Conclusion

Returning to the original question: how do you evaluate a model’s real capability in engineering scenarios? Benchmark scores are only an entry ticket. The real dividing line is this: when your instructions have gaps, does the model silently make a locally plausible decision on its own—or does it follow your design intent and make a choice consistent with the overall architecture?

Smaller models are good at executing clearly defined tasks. When boundaries are explicit, they are fast and economical. But in critical design decisions where a single omitted phrase can lead to misalignment, the intent-understanding capability of larger models is what enterprise projects are actually willing to pay for.

There is no silver bullet in model selection. Understanding the capability boundaries of each model—and codifying constraints into your workflow—is the most reliable approach for engineering teams.