Evaluating LLM Agents on Long-Horizon Tasks: Shell Execution and Self-Correction

    As AI shifts from single-turn dialogue to complex task execution, benchmarks based on real user sessions are emerging to measure what LLM agents can actually do. This article examines two key capabilities—reliable Shell command execution and mid-task course correction—through a technical lens, with a safe log-cleanup script example, a self-check checklist, and an engineering architecture (sandboxing, tracing, circuit breakers, multi-agent routing) for deploying enterprise-grade agents.

    As AI moves from single-turn conversations to executing complex tasks, LLM agents have become a major focus of the industry. One question comes up repeatedly: how much real work can a model actually do? Static question banks can measure knowledge and reasoning, but they struggle to reflect how a model performs when calling tools, writing code, and handling multi-step workflows. In response, a class of blind evaluation platforms based on real user sessions has emerged to observe how models actually behave in long-horizon tasks.

    This article does not discuss any specific product. Instead, it looks at what these evaluations measure from a technical perspective, and how those metrics can guide engineering practice.

    1. The Basic Logic of Real-Session Evaluation

    The core approach of these platforms is to collect real task sessions submitted by users, let users compare two responses without knowing which model produced them, and then compute rankings from preference outcomes. Compared with fixed question banks, the advantage is that tasks come from real needs, covering coding, office work, everyday conversation, and more.

    However, several caveats apply when reading these leaderboards. First, the user base and task distribution shape the results—a model’s ranking on coding tasks does not necessarily carry over to data analysis. Second, smaller sample sizes mean greater ranking volatility; when comparing two models, always check the sample size and confidence intervals. Third, preference votes reflect subjective user judgment, which is not the same as task completion. Leaderboards are therefore useful for spotting trends, but they should not be the sole basis for model selection.

    2. Shell Command Execution Reliability

    In programming and automated operations, agents frequently need to generate and execute Shell commands to read files, install dependencies, or run scripts. Reliability here has three layers: the command syntax is correct; the command is fault-tolerant in the target environment and does not leave a half-finished state when something goes wrong; and the command’s scope is constrained so it does not damage unrelated data.

    Consider an example that cleans up old log files. The script previews by default, requires manual confirmation before deletion, and creates a backup first:

    1. #!/usr/bin/env bash
    2. set -euo pipefail
    3. TARGET_DIR="/var/log/app"
    4. DRY_RUN="${DRY_RUN:-1}" # Preview only by default; nothing is deleted
    5. if [ ! -d "$TARGET_DIR" ]; then
    6. echo "Error: target directory does not exist: $TARGET_DIR, aborting" >&2
    7. exit 1
    8. fi
    9. # Step 1: List candidate files (top-level directory only, modified more than 7 days ago)
    10. mapfile -d '' -t FILES < <(find "$TARGET_DIR" -maxdepth 1 -name "*.log" -type f -mtime +7 -print0)
    11. echo "Candidate file count: ${#FILES[@]}"
    12. printf '%s\n' "${FILES[@]}"
    13. if [ "$DRY_RUN" = "1" ]; then
    14. echo "Preview mode: no files were deleted. Set DRY_RUN=0 to execute after review."
    15. exit 0
    16. fi
    17. # Step 2: Manual confirmation
    18. read -r -p "Confirm deletion of the files above? Type yes to continue: " answer
    19. [ "$answer" = "yes" ] || { echo "Cancelled"; exit 0; }
    20. # Step 3: Back up first, then delete
    21. BACKUP_DIR="/var/backup/app-logs/$(date +%Y%m%d_%H%M%S)"
    22. mkdir -p "$BACKUP_DIR"
    23. for f in "${FILES[@]}"; do
    24. cp -p -- "$f" "$BACKUP_DIR/"
    25. rm -f -- "$f"
    26. done
    27. echo "Cleanup complete. Backup is at $BACKUP_DIR"

    Each parameter in this code serves a purpose. set -e makes the script stop immediately if any command fails, preventing later steps from running in a broken state. set -u raises an error when an undefined variable is used, guarding against typos that could turn a path into an empty string. set -o pipefail treats a failure anywhere in a pipeline as a pipeline failure, rather than only checking the last command’s exit code.

    Paths and variables are wrapped in double quotes to prevent word splitting or glob expansion on spaces and wildcards. Using find with -print0 together with mapfile -d '' correctly handles filenames containing spaces or newlines. -mtime +7 means the file was modified more than 7 days ago—not “exactly 7 days.”

    This example only illustrates the idea. In production, additional boundaries apply. The script should run under a dedicated account with write access limited to the log directory. Deletion should have an explicit scope, without overly broad wildcards. The backup directory needs separate storage and a retention policy. For agent-generated commands, validate them in an isolated environment before running them in production.

    Improving this capability in a model typically requires introducing real terminal interaction trajectories during fine-tuning and reinforcement learning, so the model learns to read error messages and adjust subsequent commands. For example, upon seeing “Permission denied,” it should check permissions rather than retrying the same command over and over.

    3. Course Correction and Self-Correction

    Long-horizon tasks often involve dozens or even hundreds of reasoning steps. During execution, the external environment may change, files may be modified, APIs may return errors, and an early wrong judgment can amplify across later steps. Self-correction is the agent’s ability to detect deviation, stop the damage in time, and replan.

    This capability can be broken down into several checkable questions. Every few steps, run through this checklist:

    1. Goal alignment: Does the current subtask still serve the final goal? Has the agent drifted from the main thread just to complete a local step?
    2. Environment changes: Does the external state differ from the last observation? Have files, APIs, or permissions changed?
    3. Error rate: How many times has the same type of operation failed? Are the failure causes identical?
    4. Retry limit: Has the retry count for the same operation exceeded a preset cap? A hard limit is recommended—for example, retry the same command at most three times.
    5. Human takeover conditions: Has a situation requiring human intervention been triggered? Examples include deleting data, modifying permissions, sending outbound messages, or consecutive failures beyond a threshold.

    When the checklist flags an anomaly, follow this procedure: first pause write operations and perform only reads and diagnostics; record the current state and completed steps; determine whether the original path can still reach the goal—if not, generate a new plan and explain why it replaces the old one; route any high-risk operations in the new plan to human confirmation.

    This mechanism depends on the model’s context understanding, and also on prompt design and memory management. If the context grows too long, early critical constraints may be truncated, causing the agent to forget its goal. Storing the task objective, constraints, and completed steps in structured external memory is therefore usually more reliable than relying on a long context window alone.

    Recent evaluation data suggests that the gap between open-source and closed-source models on real tasks is narrowing. In scenarios such as workflow automation, open-source models are already highly competitive because they can be fine-tuned and customized. Closed-source models still hold an edge in overall performance, which is generally tied to large-scale compute and proprietary data.

    Several technical directions behind these capability gains are worth watching.

    First, the quality of tool-calling data. High-quality function-calling trajectories help models parse API responses more accurately and assemble requests that conform to format requirements. Whether a model uses parameters correctly and handles null values and error codes directly determines whether a task can proceed.

    Second, reinforcement learning from feedback. RLHF and RLAIF use human feedback and AI feedback respectively to train preference models, typically to improve response quality and safety. In multi-turn tasks, execution outcomes can also serve as reward signals—for example, whether a command ran successfully, whether tests passed, or whether the target file was generated. Such approaches require carefully designed reward functions; otherwise the model may learn to game the system, such as making tests pass without solving the real problem.

    Third, effective use of long context. In long-horizon sessions, context truncation is a common cause of task failure. Improved models compress and retrieve historical information more effectively, reducing goal drift caused by “forgetting.” On the engineering side, summarization, vector retrieval, and external memory can supplement the model’s context capacity.

    These trends should be interpreted with caution. Changes in leaderboard numbers may stem from shifts in sample composition rather than from the models themselves. When selecting a model, run small-scale tests on your own real tasks, record completion rates, the number of manual fixes required, and average time spent, and then decide whether to adopt it.

    5. Engineering Enterprise-Grade Agents

    Model capability is only the foundation. In enterprise environments, agents must run stably, which requires an engineering architecture that constrains risk, preserves records, and handles exceptions.

    5.1 Building an Isolated Execution Sandbox

    When an agent executes Shell commands or runs code, there is a risk of misoperation and data leakage. A common practice is to allocate an independent container instance per task and restrict the container’s filesystem permissions plus CPU and memory quotas. At the network level, only the egress addresses required by the task should be open, with all other access denied by default.

    Container services can be provided by compliant cloud providers or built in-house on Kubernetes. When evaluating options, focus on isolation level, audit capabilities, data storage location, and whether the service has passed security assessments.

    5.2 End-to-End Logging and Tracing

    Debugging long-horizon tasks is hard because an error may have occurred dozens of steps earlier. Log each step’s reasoning summary, tool call parameters, execution results, latency, and retry counts. Logs may contain secrets, personal information, or internal data, so they should be redacted before being written, and access should be controlled separately.

    With a complete call chain, problems can be located quickly at the specific node, without replaying the entire task from scratch.

    5.3 Monitoring, Thresholds, and Circuit Breakers

    Agents can fall into “stuck” or “infinite retry” states, wasting compute and potentially causing cascading effects. A monitoring system can enforce circuit-breaker rules—for example, if the same command fails more than N times within a fixed time window, the task is automatically paused and on-call staff are notified.

    The values of N and the time window should not be chosen by gut feel. Use historical task data or stress-test results to observe the distribution of consecutive failures in normal tasks, then set thresholds. The post-breaker procedure should also be defined in advance: preserve the scene, stop write operations, raise an alert, and resume only after human confirmation.

    5.4 Multi-Agent Collaboration and Routing

    A single agent can rarely cover all complex enterprise needs. A common architecture places a routing agent in front, which understands user intent and dispatches tasks to specialized agents for data analysis, code writing, or schedule management.

    This division of labor reduces the context pressure on any single model and makes it easier to assign different permissions to different roles. The routing layer itself needs clear logs recording the basis for each dispatch, to support later audits.

    6. Conclusion

    Agent evaluation is shifting from static question banks to real long-horizon tasks. Shell execution reliability and course-correction capability are two concrete windows into a model’s ability to act. For developers, understanding what these metrics mean is more useful than watching rankings alone.

    For adoption, start with a small pilot: choose low-risk tasks, set up preview and confirmation mechanisms, keep complete logs, and validate model performance against real task data. Expand the scope only after the results are confirmed. Rollback capability, observability, and the ability to abort are the basic prerequisites for enterprise agents to run reliably.