Selecting and Evaluating LLM Coding Assistants for Physics Simulation Development

    This article shares practical experience in using large language model (LLM) coding assistants to support physics simulation engine development. Using a lightweight kinematic solver for worm gear end-effector pose calculation as a running example, it shows how to turn a vague requirement into a verifiable, maintainable, and extensible engineering specification. It covers requirement abstraction, prompt strategy, commonalities and differences across coding assistants, an engineering approach to pure kinematic constraint solving, a multi-dimensional evaluation framework, context management, a staged iteration workflow, and risks and best practices. The core conclusion is that the engineering value of an LLM assistant should not be judged by response speed or by whether an answer merely looks correct, but by whether it helps a team reduce communication cost, shorten validation cycles, and increase constraint coverage.

    1. Background: Dynamics and Kinematics Boundaries in a Physics Simulation Engine

    During the development of an in-house physics simulation engine, the dynamics solver is usually one of the core components. Impulse-based soft-constraint dynamics approaches handle complex scenarios such as rigid body collisions, friction, contact stacking, and joint coupling reasonably well, which is why they are commonly used for high-fidelity physics simulation. However, these methods are not optimal in terms of cost for every scenario. For example, for a purely kinematically coupled linkage mechanism, sticking with the full dynamics pipeline often forces you to maintain the mass matrix, constraint Jacobian, impulse iteration, damping strategy, numerical stability, and solver parameters all at once, which makes implementation and debugging expensive.

    From an engineering perspective, this kind of need can be abstracted further: solve for the end-effector pose of a mechanism using only geometric relationships, transmission relationships, and constraint projection, without introducing full dynamics integration. This approach is better suited to lightweight scenarios such as test tools, model-authoring tools, parameter calibration tools, and rapid mechanism-validation tools. Such capabilities may not be aimed directly at end-user business features, but they matter a great deal for R&D efficiency, model debugging, and mechanism iteration.

    Therefore, when a team wants to use an AI coding assistant to quickly build this kind of non-core yet moderately complex functionality, the real question is not “which model is faster,” but rather “how do we turn a requirement into a verifiable, maintainable, and extensible engineering implementation.”

    2. Requirement Abstraction: Helping a Coding Assistant Understand “Worm Gear Mechanism End-Effector Pose Solving”

    The original requirement can be stated as: we need a capability to solve for the end-effector pose of a mechanism based on a worm gear mechanism. If you feed this single sentence directly to a coding assistant, the model may return several generalized answers, such as reusing the existing dynamics system, adding a kinematics solver module, or introducing an external solver library. These answers are not wrong in themselves, but they fall short of a solution that can actually be put into practice.

    A more sensible approach is to first break the requirement down into engineering inputs, outputs, constraints, and acceptance criteria.

    1. Inputs

    At a minimum, the following need to be specified:

    • Mechanism topology: a simple worm-and-wheel drive, or a mechanism with multiple links, closed-loop constraints, or compound transmission;
    • Driving variables: worm rotation angle, axial displacement, motor pulse count, or a normalized input angle;
    • Transmission parameters: number of worm threads, number of wheel teeth, lead, transmission ratio, reduction ratio;
    • Coordinate system definitions: base frame, end-effector frame, intermediate joint frames;
    • Boundary conditions: angle range, limits, singular positions, continuous angle handling, unit consistency.

    2. Outputs

    The output should be more than just the final pose. It should also include:

    • The end-effector pose matrix, or a combination of position and orientation;
    • Intermediate joint angles, transmission shaft angles, and axial displacement where necessary;
    • Solver status, such as whether it converged, exceeded limits, or entered a singular region;
    • Debugging information, including input-output mapping, constraint residuals, and iteration counts.

    3. Constraints

    Worm-gear mechanisms usually involve the following constraints:

    • A transmission-ratio constraint between the worm rotation angle and the wheel rotation angle;
    • A screw relationship between worm rotation and axial displacement;
    • Geometric closure constraints between links;
    • Joint angle upper and lower limit constraints;
    • Coordinate frame orientation consistency constraints;
    • Unit consistency constraints, especially conversions among angle, arc length, and linear displacement.

    4. Acceptance Criteria

    Without acceptance criteria, the code generated by a coding assistant easily stalls at the “looks like it runs” stage. More reasonable acceptance criteria include:

    • Whether the forward-solve result is consistent with the mechanism geometry;
    • Whether multiple sets of boundary inputs return stably;
    • Whether limit and extreme positions are correctly identified;
    • Whether a clear error is raised when the input exceeds the physically feasible domain;
    • Whether repeatable unit tests exist;
    • Whether explainable logs and debugging information are available.

    3. Prompt Strategy: From “Wish-Style Requirements” to “Specification-Style Requirements”

    In practice, many people default to “wish-style prompts,” for example simply saying “I need a capability to solve for the end-effector pose of a mechanism based on a worm gear mechanism.” This works for quick exploration, but it easily leads the model to give answers that look reasonable yet lack implicit constraints. The reason is that the model does not naturally understand your team’s internal code structure, coordinate frame conventions, unit conventions, mechanism parameter definitions, or existing engineering framework.

    A more robust approach is to organize the prompt as a structured specification.

    1. State the Goal

    The prompt should make clear that the current task is not to generate a complete physics engine, but to generate a lightweight kinematics solver module for rapid validation and testing.

    Example phrasing:

    “Based on pure kinematic constraints, design a small solver module. The goal is to compute the end-effector pose from the worm input angle. Do not introduce full dynamics integration.”

    2. State the Boundaries

    It is necessary to specify what should not be done:

    • Do not connect to the full dynamics system by default;
    • Do not introduce dependencies that cannot be explained;
    • Do not ignore limits and singular positions;
    • Do not output magic numbers without comments;
    • Do not assume every mechanism uses the same transmission model.

    3. State the Interface

    Interface design is key to engineering adoption. You can ask the model to output the function signature, data structures, exception types, and a usage example.

    Illustrative structure:

    1. FunctionSolveWormPose(
    2. thetaWorm,
    3. z1,
    4. z2,
    5. lead,
    6. baseTransform,
    7. offsetTransform
    8. )
    9. returns (
    10. endPose,
    11. wheelAngle,
    12. axialDisplacement,
    13. status
    14. )

    4. Ask for Tests

    You can ask the model to provide unit test ideas at the same time:

    • When the worm rotation angle is zero, the end-effector pose should equal the initial pose;
    • When the angle increases by one full cycle, the output change should match the transmission ratio;
    • When the tooth count is invalid, a parameter error should be returned;
    • When the angle exceeds the limit, a boundary status should be returned;
    • When a singular position exists, a warning should be given rather than a silent failure.

    4. Common Answers and Differences Across Coding Assistants

    For this kind of task, different coding assistants tend to give highly similar initial proposals, such as reusing the existing dynamics system, adding a kinematics solver module, or packaging it as an independent tool. This shows that at the level of general engineering understanding, coding assistants can already recognize the difference between “dynamics solving” and “kinematics solving.”

    What really deserves attention are the following dimensions.

    1. Response Speed

    Different assistants may respond at different speeds. Some tend to give answers quickly, which suits early brainstorming; others expand into longer reasoning, which suits complex solution evaluation. Response speed itself is not a drawback, but it must be judged against task complexity.

    If the task is just to generate a simple tool skeleton, speed can improve iteration efficiency; if the task involves mechanism constraints, numerical stability, and boundary handling, output that is too fast may instead miss key constraints.

    2. Reasoning Depth

    Some assistants clearly shorten their reasoning chain and quickly give a conclusion; others enumerate more boundary conditions. The advantage of the former is pace, but it may overlook implicit constraints; the advantage of the latter is robustness, but it takes longer.

    In physics simulation R&D, implicit constraints are often more damaging than explicit requirements. For example, coordinate frame orientation, angle continuity, unit conversion, singular positions, transmission ratio direction, and limit handling all affect the final result. Therefore, evaluation should not only ask “can it answer,” but “does it cover the implicit engineering conditions.”

    3. Constraint Coverage

    If a model provides only a forward formula without handling limits, exceptions, units, coordinate frames, and debugging information, then it is better suited to a concept prototype than to going directly into an engineering codebase.

    If a model can proactively raise:

    • Angle normalization;
    • Transmission ratio validity checks;
    • The relationship between axial displacement and rotation angle;
    • Coordinate frame transformation;
    • A solve-failure status;
    • Unit test suggestions;

    then it is closer to a deployable state.

    4. Maintainability

    Maintainability includes whether naming is clear, whether modules are separated, whether comments are sufficient, and whether there is unnecessary complex abstraction. For a physics simulation tool, over-abstraction is not necessarily a virtue. Clear, debuggable, and verifiable code is often more important than a formally “general architecture.”

    5. Context Cost

    When several assistants are placed in the same task, the focus of evaluation shifts from “who generates faster” to “who consumes less context, who is easier to be invoked by subsequent tools, and who is easier to fit into an automation pipeline.” If an assistant needs repeated background supplements to complete the task, its real efficiency may be lower than the impression given by its single-response speed.

    5. Engineering Approach to Pure Kinematic Constraint Solving

    For a lightweight kinematic solver, the core is not full integration but establishing constraint equations and solving for variables.

    1. General Constraint Equations

    The mechanism state variables can be defined as q, with the constraint equations:

    1. C(q) = 0

    where C represents the residual of the mechanism’s geometric, transmission, or closed-loop constraints. The goal of solving is to find the state variables q that satisfy the constraints.

    2. Jacobian Iteration

    For nonlinear constraints, an iterative method can be used. At each step, a correction is computed from the constraint residual:

    1. J = dC/dq
    2. deltaQ = -pseudoInverse(J) * C(q)
    3. q = q + damping * deltaQ

    Key points here include:

    • The way the Jacobian matrix is constructed;
    • The damping coefficient;
    • The maximum number of iterations;
    • The convergence threshold;
    • Singular matrix handling;
    • Angle unit consistency.

    3. Worm Gear Transmission Relationship

    For a worm-and-wheel drive, there is usually a rotation-angle ratio. Let the number of worm threads be z1 and the number of wheel teeth be z2, then the transmission ratio can be expressed as:

    1. ratio = z2 / z1
    2. thetaWheel = thetaWorm / ratio

    If the worm also serves a screw-advance function, then there may be a relationship between axial displacement and rotation angle:

    1. axialDisplacement = lead * thetaWorm / (2 * pi)

    where lead is the lead. In real engineering, the model must be chosen according to the specific mechanism; not all worm gear mechanisms can be reduced to the same single formula.

    4. Pseudocode Example

    The following is an illustrative structure for a pure kinematic solver, without any specific product interface:

    1. function solveWormDrivePose(thetaWorm, z1, z2, lead)
    2. require z1 > 0
    3. require z2 > 0
    4. require thetaWorm is finite
    5. ratio = z2 / z1
    6. thetaWheel = thetaWorm / ratio
    7. if lead is null
    8. return {
    9. wheelAngle: thetaWheel,
    10. status: "OK"
    11. }
    12. end
    13. axial = lead * thetaWorm / (2 * pi)
    14. return {
    15. wheelAngle: thetaWheel,
    16. axial: axial,
    17. status: "OK"
    18. }
    19. end

    The value of this illustrative code is not in its specific syntax, but in expressing an engineering principle: a lightweight kinematic solver should keep inputs, outputs, status, and exceptions as clear as possible.

    6. Evaluation Framework: Not Just Comparing Generation Speed

    When evaluating coding assistants, it is advisable to build a multi-dimensional evaluation framework rather than looking only at response speed or whether single outputs look similar.

    1. Correctness

    Correctness is the most basic metric. It includes:

    • Whether the formulas are correct;
    • Whether the transmission ratio direction is correct;
    • Whether the units are consistent;
    • Whether the coordinate frames are correct;
    • Whether the boundaries are explainable.

    2. Completeness

    Completeness focuses on whether the model covers implicit constraints:

    • Whether limits are identified;
    • Whether singularities are handled;
    • Whether angle continuity is considered;
    • Whether debugging information is output;
    • Whether the reason for failure is stated.

    3. Testability

    A good solution should be easy to test:

    • Fixed input-output samples can be constructed;
    • Extreme cases can be verified;
    • Historical mechanism parameters can be replayed;
    • Cross-validation with existing models is possible.

    4. Integrability

    A tool needs to fit into the existing R&D workflow:

    • Whether the interface is simple;
    • Whether the dependencies are heavy;
    • Whether it is easy to wrap;
    • Whether it can be called by test scripts;
    • Whether it can be reused by a GUI or a command-line tool.

    5. Context Efficiency

    In continuous development, context efficiency affects long-term cost:

    • Whether the background needs to be explained repeatedly;
    • Whether it can automatically understand context from the code repository;
    • Whether it can reduce repeated prompting;
    • Whether overly long system prompts dilute key information.

    6. Engineering Robustness

    Robustness includes:

    • Whether illegal inputs raise errors;
    • Whether numerical boundaries are controllable;
    • Whether silent failure is avoided;
    • Whether traceable logs are preserved;
    • Whether rollback is easy.

    7. Context Management and Prompt Governance

    In practice, an easily overlooked problem is context bloat. If a development tool loads too many built-in capabilities, plugin descriptions, web-reading instructions, coding standard templates, or irrelevant prompts by default, the initial context becomes too long. This not only increases response overhead, but may also scatter the model’s attention and reduce its adherence to key constraints.

    More reasonable practices include:

    1. Streamline the System Prompt

    Keep only the rules necessary for the current task, such as unit conventions, coordinate frame conventions, output format, and prohibited actions.

    2. Load Capabilities On Demand

    Do not load all tool descriptions at once. Only load the corresponding capability when the task needs to read a repository, run tests, retrieve documentation, or generate charts.

    3. Separate Stable Context from Dynamic Context

    Stable context can include project conventions, a glossary, and directory structure descriptions; dynamic context can include the current file, the latest error, and the target module. The two should be organized in layers to avoid stacking everything together.

    4. Use Summaries Instead of Full Text

    For large repositories, it is not advisable to have the model read all the code at once. It is better to first generate module summaries, interface summaries, and dependency summaries, and then have the model design the solution based on those summaries.

    5. Keep Human Review Checkpoints

    A model can generate a solution, but whether it enters the main codebase still needs developer review. Especially when physical constraints, numerical stability, and mechanism safety are involved, the final judgment should not rely on the generated result alone.

    To let a coding assistant genuinely take part in R&D rather than staying at the question-answering level, a phased iteration approach is recommended.

    Phase 1: Clarify the Solution

    Goal: get the model to understand the problem boundaries.

    Inputs:

    • The current requirement;
    • Existing physics engine capabilities;
    • The lightweight kinematic capability you want to add;
    • The constraint of not introducing dynamics integration.

    Outputs:

    • A solution overview;
    • A technical approach;
    • Risk points;
    • Information that needs to be supplemented.

    Phase 2: Design the Interface

    Goal: form a reviewable engineering contract.

    Outputs:

    • Function signatures;
    • Data structures;
    • Exception types;
    • Input-output examples;
    • Invocation approach.

    Phase 3: Minimal Implementation

    Goal: generate a runnable skeleton.

    Outputs:

    • Core solving logic;
    • Parameter validation;
    • Status returns;
    • Basic logging.

    Phase 4: Fill in Tests

    Goal: verify correctness.

    Outputs:

    • Positive tests;
    • Boundary tests;
    • Exception tests;
    • Extreme position tests;
    • Unit conversion tests.

    Phase 5: Integration and Observation

    Goal: integrate into the R&D environment.

    Outputs:

    • The module entry point;
    • Test scripts;
    • Runtime logs;
    • Performance observation points;
    • Follow-up optimization suggestions.

    This workflow is more reliable than a single “wish-style generation,” because each step has clear inputs, outputs, and acceptance criteria.

    9. Risks and Best Practices

    1. The Model Is Not a Domain Expert

    Coding assistants can help generate code and solutions, but for the physical meaning of a specific mechanism, assembly errors, friction, clearance, material deformation, and so on, they cannot inherently give fully reliable judgments. Key conclusions still need to be confirmed by engineers.

    2. Similar Answers Do Not Mean Correct

    Multiple assistants giving similar answers only shows that the problem is relatively clear at the general level; it does not prove that the implementation details are correct. Testing, simulation comparison, and engineering review are still required.

    3. Implicit Constraints Must Be Made Explicit

    Many problems in physics simulation come from implicit constraints. Both prompts and code comments should explicitly record:

    • Units;
    • Coordinate frames;
    • Positive and negative directions;
    • Angle ranges;
    • Limit strategies;
    • Singularity handling;
    • Transmission ratio direction;
    • Initial state assumptions.

    4. Do Not Let Generated Code Enter the Core Pipeline Directly

    For non-core test tools, the model can be allowed to quickly generate a prototype; for the core physics engine, strict review should be required. Even if the final code is written or rewritten by a human, the model can still take on auxiliary work such as drafting the solution, designing test cases, designing logs, and suggesting refactors.

    5. Establish a Regression Baseline

    Once the mechanism model is usable, fixed input-output samples should be established as a regression baseline. Any subsequent model upgrade, code refactor, or parameter adjustment can then be validated against these samples.

    10. Conclusion: Rationally Evaluating the Engineering Value of Coding Assistants

    In physics simulation engine R&D, evaluating coding assistants should not look only at whether a single answer is fast, nor only at whether the answer “looks correct.” A more reasonable evaluation approach is to observe whether the assistant can help the team turn vague requirements into clear specifications, turn implicit constraints into explicit code, and turn one-time generation into an iterable, testable, and maintainable engineering process.

    From a practical standpoint, a lightweight reasoning mode is suited to rapid exploration but may miss implicit constraints; a deep reasoning mode is more robust but takes more time. The genuinely valuable approach is to choose different modes according to the task phase: use the lightweight mode for early exploration, and the deep mode for key constraint design, complex mechanism derivation, test plan completion, and code review.

    In the end, the demonstration of a model’s capability lies not in whether it can give an answer that looks reasonable, but in whether it can help developers reduce communication cost, shorten validation cycles, increase constraint coverage, and make the final solution easier for an engineering team to accept and maintain over the long term.