Post-Training Optimization and Long-Horizon Agent Engineering: A Practical Guide

    This guide covers two core engineering topics for LLM practitioners: how to run post-training with verifiable rewards—including reinforcement learning frameworks, sandbox infrastructure, and million-token context memory budgeting—and how to validate model-generated project skeletons in code engineering, with concrete acceptance criteria, risk controls, and a runnable scaffolding example.

    Post-Training Optimization: Start by Defining Verifiable Rewards

    The core of post-training is making the model continuously better at verifiable tasks. Every piece of work should clearly define four things: the task type, the input, the acceptance criteria, and the risks.

    1. Reinforcement Learning Framework: From Sparse Rewards to Process Rewards

    • Task types: code repair, mathematical reasoning, tool calling.
    • Inputs: task descriptions, initial codebases, or environment snapshots.
    • Reward sources: unit test pass rates, compilation success, static analysis results, output format validation.
    • Process rewards: verifiable signals for intermediate steps in multi-step reasoning—for example, whether a tool call returned the expected structure, or whether a patch passes the relevant subset of tests. A process reward model needs to be evaluated separately for its correlation with final outcomes. When correlation is low, it should not be used directly for training.
    • Acceptance criteria: maintain a held-out test set that never participates in training. If training-set rewards keep rising while held-out scores stay flat, the model is likely gaming the metric—reward hacking.

    Three safeguards apply. First, periodically sample and audit high-reward trajectories to check whether the model is scoring points by modifying tests, deleting assertions, or hardcoding outputs. Second, monitor response length to prevent the model from inflating scores with verbose output. Third, use KL constraints to limit how far the policy drifts from the baseline model.

    2. Sandbox Infrastructure and Environment Quality Checks

    Code executed by an agent should be treated as untrusted input. The sandbox must meet the following constraints:

    Dimension Recommended Practice
    Isolation Use a separate container or microVM per task, running as a non-root user
    File system Read-only root filesystem; only the task directory is writable; destroyed after the task ends
    Network Offline by default; egress only through whitelisted domains when needed
    Resources Set limits on CPU, memory, disk, and per-task timeout
    Credentials Never inject production keys; use short-lived, least-privilege tokens
    Logging Redact sensitive fields such as phone numbers, ID numbers, and secrets before writing

    Environment quality checks confirm that feedback signals are reliable. You can track the following metrics and set thresholds against your own baseline: environment startup success rate, P95 startup latency, the consistency rate of running the same test three times, and environment reset time. Test cases that produce inconsistent results are flaky cases. Unstable environments write noise into the reward signal and should be fixed or removed first.

    3. Million-Token Context: Memory Accounting and Scheduling

    KV cache memory usage scales linearly with sequence length. The bytes per token can be estimated as:

    KV bytes/token = 2 × number of layers × number of KV heads × head dimension × bytes per element

    Take a hypothetical configuration: 80 layers, 8 KV heads, head dimension 128, stored in FP16. A single token occupies about 327,680 bytes, roughly 320 KiB. For a single sequence of one million tokens, the KV cache alone is about 328 GB—more than the memory of a typical single GPU. In practice, you usually need to combine several of the following:

    • Use Grouped-Query Attention (GQA) or Multi-Query Attention to reduce the number of KV heads.
    • Manage the KV cache with paged attention to reduce memory fragmentation.
    • Enable prefix caching to reuse shared context across requests.
    • Quantize the KV cache to FP8 or INT8. After quantization, regression-test long-range retrieval accuracy.
    • Split long-sequence computation with sequence parallelism or tensor parallelism.

    The boundaries of applicability must be clear. Time to first token (TTFT) grows noticeably with input length in long-context scenarios, so real-time interactive use cases should cap input length or switch to segmented summarization. Long-range retrieval capability should be validated with needle-in-a-haystack tests at different positions and different lengths—looking only at average scores is not enough.

    Code Engineering: How to Validate Model-Generated Project Skeletons

    Code engineering can be divided into two granularities. The small granularity is fixing a single issue in a real codebase; the large granularity is generating a complete project from requirements. The two require different acceptance approaches.

    Task Type Input Tools Acceptance Criteria Main Risks
    Single-point fix Issue description, failing tests, related files Code search, test runner Original failing tests pass; no new failures in the full test suite Modifying the tests themselves to pass
    Project skeleton Requirements doc, tech stack, directory conventions Scaffolding scripts, package manager Directory structure matches conventions; project installs and passes empty tests Path escapes, overwriting existing files
    API documentation Interface definitions, sample requests Documentation generators Docs match the interface signatures in code Documentation inconsistent with implementation

    Directory names and file contents produced by the model should be parsed and validated first, then handed to deterministic scripts for writing. The model itself never executes commands directly. Here is a minimal runnable example:

    ```python
    import re
    import shutil
    from pathlib import Path

    TEMPLATES = {
    “python-cli”: [“src”, “tests”, “docs”],
    }
    NAMERE = re.compile(r”[a-z][a-z0-9]{1,31}”)

    def scaffold(name: str, base_dir: str, kind: str = “python-cli”) -> Path:
    if not NAME_RE.fullmatch(name):
    raise ValueError(“Project name must start with a lowercase letter, contain only lowercase letters, digits, and underscores, and be 2 to 32 characters long”)
    if kind not in TEMPLATES:
    raise ValueError(f”Unsupported template: {kind}”)

    1. # The name has passed regex validation and contains no path separators,
    2. # so root is guaranteed to stay inside base
    3. base = Path(base_dir).resolve()
    4. root = base / name
    5. if root.exists():
    6. raise FileExistsError(f"Directory already exists, refusing to overwrite: {root}")
    7. try:
    8. root.mkdir(parents=True)
    9. for sub in TEMPLATES[kind]:
    10. (root / sub).mkdir()
    11. pkg = root / "src" / name
    12. pkg.mkdir(parents=True)
    13. (pkg / "__init__.py").write_text('"""Project entry package."""\n', encoding="utf-8")
    14. (pkg / "main.py").write_text(
    15. 'def main() -> None:\n print("hello")\n\n\nif __name__ == "__main__":\n main()\n',
    16. encoding="utf-8",
    17. )
    18. (root / "README.md").write_text(