Engineering Evaluation of Code LLMs in Physics Simulation Engine Development

    This article explains how to evaluate code large language models on a practical physics engine task: adding a lightweight kinematic constraint solver alongside an existing dynamics solver. It covers how to turn a vague feature request into verifiable requirements, why first-round answers often miss hidden constraints, and how to measure correctness, iteration cost, and context cost.

    1. Background: Dynamics Solving Is Not the Default for Every Task

    In the development of an in-house physics simulation engine, the impulse-based soft constraint dynamics solver usually handles the core simulation work. Its handling of contact, collision, friction, constraint compliance, and time stepping determines how credible a complex scene is. However, being able to simulate does not mean every toolchain should reuse the same solving path.

    For purely kinematic linkages, such as worm drives, gear couplings, closed-loop linkages, assembly fixtures, and test prototypes, the task usually cares only about geometric pose relationships, transmission ratios, and assembly constraints. It does not care about real contact forces or energy dissipation. Running these cases through the full dynamics soft constraint solver adds unnecessary state variables, iteration cost, and parameter tuning effort.

    A reasonable engineering requirement follows from this. Without changing the existing dynamics path, add a lightweight, independently testable pure kinematic constraint solver. It would serve model making, mechanism inspection, pose verification, and test tools. This feature is not part of the engine core, but it works well as a benchmark sample for code LLM-assisted development. It has a clear business context and measurable correctness criteria, and it contains many implicit constraints that only show up in a real codebase.

    2. Turning a Wish-Style Request into a Verifiable Task

    An initial request might be as simple as “Given a worm mechanism, solve for the end-effector pose” or “Support end-effector pose solving for worm-driven mechanisms.” If the model receives this wish directly, a common result is that it reuses the existing dynamics solver and presents a plausible-looking solution. The problem is not that the model cannot write the code. The problem is that the task has not been defined as an engineering task.

    Before evaluation, the requirements should be split into at least the following categories:

    1. Functional inputs and outputs: The inputs are the mechanism topology, joint types, transmission ratios, and a target end-effector pose or target joint state. The outputs are joint variables that satisfy the constraints, the end-effector pose, the residual, and the failure reason.
    2. Boundary conditions: The existing dynamics solving path must not be modified. The new module must compile and test independently. It must not introduce uncontrolled external dependencies.
    3. Constraint types: Fixed constraints, closed-loop constraints, gear or worm coupling constraints, joint limits, target pose constraints, and singularity handling where needed.
    4. Numerical metrics: Residual convergence threshold, maximum iteration count, fallback strategy on failure, determinism, thread safety, and performance limits.
    5. Acceptance criteria: Unit tests that cover typical mechanisms, extreme transmission ratios, singular configurations, and random initialization. Consistency checks of the Jacobian against finite differences. Comparison with existing high-precision reference results.

    The problem can be formalized. Let the generalized coordinates q contain joint displacements or rotation angles, and let the target end-effector pose be T_target. The constraint set can be written as:

    • Fixed and closed-loop constraints: C_fix(q)=0;
    • Transmission coupling constraints: C_couple(q)=0, where the worm and worm gear transmission ratio, phase, direction, and zero position must be specified;
    • Target pose constraint: C_pose(q)=log(T_target^-1 * T_ee(q))=0;
    • Joint limits: q_min <= q <= q_max.

    This step matters because it changes the question from “Can the model write this piece of code?” to “Can the model deliver something verifiable within real engineering boundaries?” Without this definition, a fast answer can satisfy the surface request while breaking implicit requirements such as lightweight design, non-intrusiveness, and testability.

    3. First-Round Results: Reuse Bias and Implicit Constraints

    In first-round comparisons on similar tasks, candidate models often proposed reusing the existing soft constraint solver and tuning its parameters. The approach looks reasonable on the surface. It reuses mature code, adds few new modules, and lowers short-term risk. However, it may conflict with the actual goal of providing a lightweight path for pure kinematic scenarios. If a model makes only local completions based on the existing implementation, it can easily mistake “it runs” for “it is acceptable.”

    Another pattern is the trade-off between generation speed and robustness. Faster answers usually come from shorter reasoning chains and fewer clarification steps. This helps with exploration and draft generation. However, fast answers more often overlook implicit constraints such as units, coordinate conventions, transmission sign, joint limits, singular configurations, determinism, and regression tests. A single-turn output may look complete, but it can require several rounds of rework once it is placed in a real engineering project. In contrast, a solution that reasons longer and describes its assumptions more conservatively may be slower, but it tends to cover failure modes, boundary conditions, and test recommendations more completely.

    Therefore, evaluating code LLMs should not stop at whether the first-round answer looks acceptable. A more sound conclusion is this. Short reasoning chains suit fast draft generation, but they must be paired with structured review and automated tests. Longer reasoning chains suit complex constraint completion, but their interaction cost must be controlled. The comparison should focus on deliverable quality after multiple iterations, regression stability, and total cost of ownership, not on the impression created by a single question and answer.

    4. Reference Implementation of a Lightweight Pure Kinematic Constraint Solver

    A practical implementation should keep the module independent. The kinematic solver should depend only on pose computation, constraint residuals, and Jacobians. It should not depend on dynamics state updates. The core solver can use damped least squares iteration. Its pseudocode is shown below:

    1. function solveKinematic(state, constraints, targetPose, options):
    2. q = state.q
    3. for iter in 1..options.maxIter:
    4. C = []
    5. J = []
    6. for constraint in constraints:
    7. residual, jacobian = constraint.residualAndJacobian(q)
    8. C.append(residual)
    9. J.append(jacobian)
    10. poseResidual, poseJacobian = poseResidualAndJacobian(q, targetPose)
    11. C.append(poseResidual)
    12. J.append(poseJacobian)
    13. concatenate C and J
    14. if norm(C) < options.tolerance:
    15. return success(q, C)
    16. H = transpose(J) * W * J + options.damping * I
    17. dq = solveLinearSystem(H, -transpose(J) * W * C)
    18. step = lineSearch(q, C, dq)
    19. q = integrate(q, step)
    20. q = clampJointLimits(q)
    21. return residualTooLarge(q, C)

    Several details require special attention during implementation:

    • Constraint grouping: Compute residuals and Jacobians separately for fixed, closed-loop, transmission, and target pose constraints. This makes it easier to locate the source of a failure.
    • Jacobian verification: Sample each constraint’s Jacobian and check it against finite differences to catch sign errors.
    • Singular configurations: Use damping, step size limits, and residual-based fallback to prevent numerical divergence.
    • Redundant degrees of freedom: Add a joint-center preference or a minimum displacement preference in the null space.
    • Units and coordinate frames: Declare length units, angle units, rotation order, and reference frames explicitly at the interface.
    • Failure semantics: Distinguish states such as residual over the limit, conflicting joint limits, and unsatisfiable constraints. Do not return an empty result for all of them.

    The boundary between this solver and the dynamics soft constraint solver should also be documented clearly. The kinematic module handles only geometric and transmission relationships. It does not simulate contact forces, friction, inertia, or energy dissipation. It suits model making and kinematic verification. It should not be mistaken for a replacement for dynamics.

    5. An Engineering Approach to Evaluating Code LLMs

    To turn this kind of task into a reproducible evaluation, build a lightweight benchmark set rather than relying on subjective impressions. The evaluation should cover at least the following dimensions:

    1. Correctness: Pass rate on hidden test cases, especially for units, transmission sign, closed-loop constraints, and extreme configurations.
    2. Constraint coverage: Whether the model proactively identifies implicit boundaries in a non-core goal, such as non-intrusiveness, testability, and no external dependencies.
    3. Maintainability: Whether the interface is clear, whether error handling is complete, and whether tests and documentation are included.
    4. Iteration efficiency: The number of rounds needed to reach an acceptable result, the amount of manual correction, and the number of regression runs.
    5. Resource cost: Context length, generation latency, retry count, and the overhead of tool descriptions.
    6. Stability: Whether repeated runs of the same task produce solutions with a consistent structure, and whether random drift occurs.

    For collaboration, have the model write a design note before writing code. The design note should include at least the module boundary, inputs and outputs, a constraint list, failure modes, a test plan, and performance assumptions. Then proceed in this order: interface definition, core solver, unit tests, integration verification, and optimization. Compared with asking for a complete implementation in one pass, this staged approach makes misunderstandings easier to spot and makes human review easier.

    Context cost matters as well. Some engineering tools enable large amounts of tool descriptions, retrieval instructions, and system prompts by default, and these consume a large share of the context in every request. An evaluation should not look only at the length of the user input. It should record the total consumption of system prompts, tool definitions, retrieved snippets, conversation history, outputs, and retries. For simulation engine subtasks, structured summaries, on-demand retrieval, and a reduced tool set are usually more economical than attaching everything.

    6. From Model Evaluation to Engineering Assets

    For engineering teams, this kind of evaluation can become an internal asset:

    • Turn typical mechanisms, extreme parameters, and failure cases into regression tests.
    • Design the kinematic solver as an independent module with a clear interface, so it can connect to modeling tools and test tools.
    • Run residual, Jacobian, and pose accuracy tests in continuous integration to prevent model-generated code from breaking the existing path.
    • Record the context consumption, iteration rounds, and manual correction time of each approach. Select models based on engineering metrics rather than single-response impressions.
    • Treat model output as a draft. Final acceptance should come from automated tests and human review together.

    Overall, the value of code LLMs in physics simulation engine development does not lie in whether one answer looks like an expert’s. It lies in whether they can quickly produce verifiable, maintainable, and regression-safe engineering increments within clear boundaries. Short reasoning chains can speed up exploration, but tests and constraint completion must back them up. Multi-round collaboration can approach a usable solution, but context and repeated work must be kept under control. Requirement definition, implicit constraints, numerical verification, and cost accounting should all sit in one process. That is the reliable way to evaluate this kind of capability.