Full-Parameter Reinforcement Learning for Optimizing AI Agent Performance and Cost Under Limited Compute
As large language models shift from single-turn question answering to multi-step agentic tasks, optimizing them increasingly means balancing task success rate, execution efficiency, and per-task cost rather than simply scaling up model size. This article explains why full-parameter reinforcement learning helps agents learn more effective execution strategies, how freezing non-critical modules improves training stability, and how reward design, cluster-level engineering practices, inference optimization, and multi-dimensional evaluation work together to achieve a sustainable performance-cost balance on limited compute. It is aimed at engineers and teams building or optimizing agent systems.
Large language model (LLM) applications are shifting rapidly from single-turn question answering to multi-step task execution. Agents no longer just output a piece of text; they need to understand goals, break down steps, call tools, read intermediate results, handle exceptions, and complete tasks under limited resources. This shift fundamentally changes the optimization objective: the focus used to be answer quality, but now it also covers task success rate, execution stability, context utilization efficiency, the number of tool calls, the number of failure retries, and total cost per task.
Under limited compute, simply scaling up model size does not always help. A larger model may improve performance on some complex tasks, but it also brings higher inference costs, longer response times, and less stable resource consumption. A more practical engineering path is therefore not to pursue “bigger” for its own sake, but to seek Pareto improvements around a specific task set: raising task success rate without significantly increasing cost, or lowering per-task cost while keeping performance roughly unchanged.
1. Why performance and cost must be optimized together under limited compute
Agent tasks differ significantly from chat tasks. Chat tasks can usually be evaluated offline, and output quality relies heavily on human or automated scoring. Agent tasks, in contrast, often come with verifiable goals — for example, whether code compiles, whether tests pass, whether files are generated, whether commands return the expected results, and whether the process is reproducible. These tasks are better suited to a closed-loop evaluation system.
In this setting, model cost is not determined by parameter count alone. A single task execution may involve multiple inference rounds, multiple context segments, multiple tool calls, retries of subtasks, and even multiple environment instances running in parallel. If a model tends to call tools repeatedly, generate verbose explanations, or fail to locate errors quickly, it can consume large amounts of resources even when it eventually succeeds. Cost evaluation should therefore cover input and output tokens, the number of inference rounds, the number of tool calls, environment execution time, cache hit rate, and failure-retry overhead.
From an optimization perspective, three core metrics can be introduced:
- Task success rate: reflects whether the model actually completes the task.
- Execution efficiency: reflects whether the path the model takes to complete a task is concise and stable.
- Per-task cost: reflects the resources a model consumes in a single task.
The engineering goal is usually not a single-point optimum, but finding a better set of parameters and strategies that strike a more reasonable balance among success rate, efficiency, and cost.
2. The value of full-parameter reinforcement learning in agent optimization
The core value of full-parameter reinforcement learning (RL) is that the model can adjust all language-side parameters based on task execution feedback, rather than only tuning a few adapter layers. For agent tasks, what the model needs to learn is not just “generating correct text,” but also “choosing the right action at the right time,” “adjusting its strategy after a failure,” and “repairing things when a tool returns an exception.” These abilities are often difficult to acquire through supervised fine-tuning alone.
RL converts task outcomes into optimization objectives through reward signals. With a well-designed reward, the model can gradually learn more efficient execution paths — for example, reducing ineffective steps, avoiding repeated calls, prioritizing verification of key results, and confirming in small steps when uncertain. For long-chain tasks, this approach is more systematic than simply adding more prompt templates.
Full-parameter updates also come with clear challenges. The first is memory pressure: optimizer states, gradients, activations, and inference caches all consume substantial resources. The second is training stability: task rewards may be noisy, and if the difficulty distribution of samples is uneven, training can oscillate. The third is engineering complexity: agent training requires simultaneously maintaining a policy model, an execution environment, a reward model, a data sampler, and a monitoring system.
Full-parameter RL is therefore not suitable for every scenario. More ideal prerequisites are clear task goals, verifiable results, a controllable execution environment, a moderate data scale, and a team with complete distributed training and experimental evaluation capabilities.
3. Freezing non-critical modules to improve stability under limited resources
When training large models on 100-GPU-scale clusters or smaller setups, resource allocation must be precise. Not all modules in a model need to be updated at the same time. A common practice is to freeze the parts that are weakly related to the current optimization goal and train only the key language parameters and auxiliary prediction modules.
For example, a visual encoder is usually used to extract image or multimodal features. If the current training goal focuses on text and tool tasks such as code generation, terminal operations, and process planning, updating the visual encoder has relatively limited value. Freezing the visual part reduces memory usage, lowers data requirements, and avoids drift in multimodal representations during text tasks.
Retrieval-index modules should also be updated cautiously. If a system relies on an external knowledge base or on tool return results for reasoning, changes to the index will affect task reproducibility. Freezing the indexer decouples the training process from the retrieval state, making it easier to determine whether a problem comes from the model policy or from changes in retrieval results.
In addition, modules such as multi-token prediction, auxiliary heads, and draft heads can sometimes participate in training, because they help improve planning ability and output continuity. However, these modules also increase training complexity, so tradeoffs need to be decided based on task benefit and resource cost. The basis for freezing or unfreezing should not be “training all modules gives the best results,” but rather “which modules maximize task benefit while minimizing training risk.”
4. Key engineering practices for 100-GPU-scale cluster training
Completing a round of post-training on a cluster of tens of accelerators is often a challenge not just in the algorithm, but in whether the engineering system can stably support long-running experiments. A few key aspects are involved.
1. Parallelism strategy design
Large model training usually involves multiple forms of parallelism, including data parallelism, tensor parallelism, pipeline parallelism, and sequence parallelism. Which combination to choose depends on the model structure, the number of activated parameters, sequence length, memory capacity, and interconnect bandwidth. An unreasonable parallelism strategy can create communication bottlenecks that reduce accelerator utilization.
In practice, you can first determine memory usage and throughput with small-scale prototype experiments, then gradually scale up to the full cluster. For long-context tasks, sequence splitting and attention computation are especially important; for tasks with long tool calls and intermediate states, you also need to pay attention to context caching and activation management.
2. Data sampling and task-difficulty stratification
The effectiveness of RL depends heavily on data quality. The task set should cover different difficulty levels, styles, and failure modes. If samples are too simple, the model cannot learn effective strategies; if they are too difficult with sparse rewards, training tends to stall.
Sampling can be stratified by difficulty — for example, basic tasks, composite tasks, exception-recovery tasks, and long-chain tasks. Increase the sampling weight for high-failure-rate tasks, and moderately reduce the proportion of tasks the model has already mastered stably. Data deduplication is also important: repeated tasks bias the reward signal and waste inference and execution resources.
3. Execution sandbox and tool environment
Agent training usually involves command execution, file read/write, network requests, or API calls. It must run in an isolated environment to avoid affecting the main training cluster. The sandbox should provide resource limits, network limits, logging, and exception recovery.
Environment execution should be as automated as possible, and complete trajectories should be preserved. The input, tool calls, return results, error messages, elapsed time, and resource consumption of each task execution should be recorded. This data is both the source of the training reward and the basis for subsequent problem analysis.
4. Checkpointing and fault tolerance
Once 100-GPU-scale training is interrupted, the loss can be significant. The system should support periodically saving the model, optimizer state, and training progress. For abnormal tasks, rather than simply discarding them, you should classify and record them — environment faults, model failures, tool timeouts, reward anomalies, and so on — since different problems call for different handling.
During training, you also need to monitor loss changes, mean reward, task success rate, gradient norm, memory usage, throughput, and queue backlog. If metrics fluctuate abnormally, you should pause promptly and investigate the data distribution or reward design.
5. Reward design: improving performance while suppressing cost
Reward design in agent tasks is more complex than in traditional RL. Task outcomes are often multi-stage, and relying only on final success or failure can produce a sparse signal. It is usually necessary to combine outcome rewards with process rewards.
Outcome rewards focus on whether the final goal is achieved — for example, whether tests pass, whether the output meets requirements, and whether files are runnable. Process rewards focus on step quality — for example, reducing ineffective tool calls, correcting errors promptly, and avoiding over-generation.
A simplified reward approach looks like this:
for step in range(total_steps):tasks = sampler.sample(batch_size)trajectories = policy.rollout(tasks)outcomes = executor.run(trajectories)task_success = verify(outcomes)quality_score = evaluator.score(outcomes)cost = estimate_cost(trajectories)reward = (0.7 * task_success+ 0.3 * quality_score- 0.05 * cost)update_policy(policy, trajectories, reward)
Here, the cost term can be used to suppress verbose reasoning and excessive tool calls. One important caveat is that cost should not over-suppress exploration. If the cost penalty is too strong, the model may tend to avoid complex tasks or terminate too early. It is therefore advisable to set thresholds or use layered rewards, applying the penalty only when cost deviates significantly from a reasonable range.
You should also guard against reward hacking. For example, a model might generate output that looks complete but is actually ineffective in order to earn a score, or repeatedly call certain low-cost tools to create the illusion that it is “working.” To address this, the reward function should rely as much as possible on verifiable results, combined with manual spot checks, rule-based validation, and anomaly-pattern detection.
6. From training to inference: systematically optimizing per-task cost
Reducing per-task cost cannot rely on the training stage alone; the design of the inference stage is just as important. Here are several general optimization techniques.
1. Context compression
In agent tasks, the content returned by tools can be long. If it is all retained, it quickly consumes context and raises cost. Historical information can be summarized, trimmed, structurally extracted, and deduplicated, keeping only the information relevant to the current step.
2. Caching
For repetitive requests, identical tool calls, similar queries, and stable template outputs, you can build caches. Caching not only reduces token consumption but also shortens response time. However, cache-key design must be careful to avoid incorrectly reusing results across different tasks, permissions, or time states.
3. Model routing
Not every task needs to be handled by the largest model. Simple tasks can be assigned to a smaller or faster model, while complex tasks can be escalated to a stronger model. With a task classifier or rule-based routing, you can lower the average cost while preserving performance.
4. Batching and asynchronous execution
For tasks that can run in parallel, you can increase the batch size to improve resource utilization. For long-chain tasks, some steps can be executed asynchronously to reduce waiting time. Although batching may slightly affect single-task latency, it usually significantly reduces unit resource cost.
5. Tool-call convergence
Models should prioritize high-certainty tools and avoid repeated calls. You can train the model to learn tool-selection strategies, or add call-count limits, duplicate detection, and execution-result reuse at the system level.
7. Evaluation: how to prove that optimization is effective
Model optimization under limited compute cannot rely on a single metric. A multi-dimensional evaluation system is needed, covering performance, cost, stability, and reproducibility.
1. Performance metrics
Common metrics include Pass@1, Pass@k, averages over multiple trials, task completion rate, test pass rate, and human ratings. For agent tasks with high randomness, you should repeat experiments multiple times and report both the mean and the variance, to avoid misjudgments caused by random fluctuation in a single run.
2. Cost metrics
Cost should be defined as the average consumption per task, including model input and output, tool calls, environment execution, cache hits, and failure retries. If a model is deployed by the team itself, costs should be converted based on actual resource usage; if public service prices are used as a reference, the billing basis should be stated to avoid making different approaches incomparable.
3. Pareto analysis
You can plot task success rate against per-task cost as a scatter plot to see where different configurations sit. A genuinely valuable optimization shows points moving in a better direction: success rate rises without a significant increase in cost, or cost falls while success rate stays roughly the same.
4. Failure analysis
Evaluation is not only about scoring, but also about discovering problems. You should build a failure classification system — for example, understanding errors, planning errors, tool-selection errors, environment anomalies, context loss, and reward bias. Different failure types point to different optimization directions: understanding errors require improvements to data and prompting strategies, tool-selection errors require adjustments to rewards and training samples, and environment anomalies require engineering governance.
8. Security, compliance, and observability
Agent tasks often involve file operations, command execution, and tool calls, so security boundaries must be designed up front. Systems should follow the principle of least privilege and grant only the permissions a task needs. High-risk operations should have a confirmation mechanism, and critical resources should be managed in isolation.
Logging and auditing are equally important. Every training task, tool call, environment execution, and model output should be traceable. For sensitive data, it should be desensitized or permission-checked before entering the training set. Keys, credentials, and access tokens should never appear in context, logs, or training data.
For observability, it is advisable to build a unified monitoring view that includes training metrics, inference metrics, environment metrics, and cost metrics. On the training side, focus on loss, reward, success rate, and throughput; on the inference side, focus on latency, token consumption, cache hit rate, and failure rate; on the environment side, focus on sandbox resources, execution time, and anomaly frequency.
If an enterprise runs on a cloud platform, it can use general cloud capabilities to host these components. For example, object storage can hold training data, model checkpoints, and task trajectories; a container platform can host sandboxes and inference services; a log service can aggregate training and call logs; and monitoring and alerting can surface resource bottlenecks and anomalous trends. These capabilities help turn model training from a one-off experiment into an engineering system that can be iterated continuously.
9. Implementation recommendations
For teams that want to reproduce similar optimization, the following steps can be followed:
- Clarify the task scope, prioritizing task sets with verifiable goals and clear feedback loops.
- Establish a baseline model and a baseline cost, so that subsequent optimization has a clear reference.
- Design a lightweight reward function, starting with rule-based rewards and verifiable signals for experiments.
- Validate training stability at a small scale first, then gradually expand cluster and data scale.
- Model training and inference costs separately, to avoid focusing only on performance while ignoring long-term operating overhead.
- Build a failure-feedback mechanism that turns online failure tasks into training samples.
- Continuously perform Pareto evaluation, rather than pursuing improvement in a single metric.
Overall, large model optimization under limited compute should not be understood as simply “training a bigger model,” but as systematic optimization around the task closed loop. By using full-parameter RL to improve policy quality, freezing non-critical modules to control resource consumption, balancing performance and cost through reward design, and lowering per-task cost through inference optimization and engineering governance, it becomes possible to achieve a more sustainable balance between performance and cost.
