Local-First Agent Framework: Building AI Agents That Actually Do Work

“AI that can actually do work” is a phrase that has appeared frequently in the AI agent space in recent years. It refers to systems that can autonomously plan steps, call tools, read and write files, execute commands, operate business systems, and complete end-to-end tasks. Compared with chatbots that only generate text, this kind of agent is closer to a “digital colleague” that can deliver results.

This article uses “local-first” as the starting point to systematically explain the definition, architecture, and key design points of such agents. It also provides a practical minimal framework. The focus is on how to move AI from “talking” to “doing,” while keeping data and execution controllable.

1. From Chatting to Doing Work

The original capability boundary of large language models was mainly text generation: they could answer questions, summarize content, and write code snippets, but they could not directly sense or change the external environment. Agents break through this boundary by adding three key capabilities:

1. Perception and action capabilities: The model can read and write files, execute commands, send requests, and query databases through tool calls, turning reasoning results into real actions.
2. Autonomous decision loops: The model decides the next step based on the result of the previous step. It can observe, adjust, and correct itself instead of producing a one-time answer.
3. Clear permission boundaries: Once actions can actually take effect, constraints are required. Otherwise, “being able to do work” quickly becomes “being able to cause damage.”

Therefore, the standard for judging whether something is “really capable of doing work” does not lie in the model itself, but in the engineering system around the model. A typical agent framework usually includes the following layers:

┌─────────────────────────────────────────────┐
│                Interaction Layer            │
│   CLI / desktop client / IDE plugin / business system   │
├─────────────────────────────────────────────┤
│             Orchestration Layer (Agent Loop)             │
│      Task planning · step control · state and budget management   │
├──────────────┬──────────────┬───────────────┤
│   Model Layer  │   Tool Layer   │   Memory Layer   │
│ Model access and routing │ Registration, validation, and dispatch │ Context, history, and knowledge │
├──────────────┴──────────────┴───────────────┤
│           Execution and Permission Layer           │
│      Sandbox · permission engine · human approval · audit       │
├─────────────────────────────────────────────┤
│              Infrastructure Layer               │
│     Local compute resources / model services / logs / monitoring  │
└─────────────────────────────────────────────┘

2. What “Local-First” Really Means

“Local-first” does not simply mean “installing the program on your own computer.” It is a design principle for data and execution. It can be divided into three meanings:

• Data layer: User files, code, and business data are stored locally or in an enterprise-owned environment by default. Only the content necessary to complete a task is sent to model services when needed, and the process should be auditable and configurable.
• Execution layer: Commands, scripts, and file operations are executed in a controlled local sandbox, and results are written directly back to the user’s own working directory. This reduces the cost of repeatedly moving files between the cloud and local machines.
• Governance layer: Users fully control how models are connected, what tools are available, and what permission policies are enforced. Whether a cloud model service or a locally deployed model is used, the setup can be adjusted or replaced at any time.

Local-first agents and pure cloud agents each have their strengths:

Dimension	Local-First	Pure Cloud
Data residency	Local or enterprise environment by default	Mainly in the cloud
Offline and weak-network availability	Better, especially when local models are used	Worse
Local resource access	Direct and low latency	Requires additional integration
Elastic compute	Limited by local hardware	Strong elasticity
Operational complexity	Higher because deployment is required on the user side	Lower

The two approaches are not mutually exclusive. In practice, a hybrid model is more common: sensitive data and execution processes remain local, while complex reasoning tasks call cloud model services when needed. For example, task logs and artifacts generated by an agent can be stored in object storage for later auditing and analysis. General capabilities provided by cloud platforms, such as those offered by Baidu AI Cloud, can be used for these peripheral components while preserving the design principle of keeping core data and execution local.

3. A Minimal Working Framework: Agent Loop and Tool System

3.1 The Agent Loop: The Heart of an Agent

The essence of an agent loop is a repeated process of “think, act, observe.” Below is a minimal implementation for illustration:

def run_agent(task, ctx, budget):
    while budget.steps_left():
        budget.charge_step()
        # 1) Think: the model decides the next step based on the current context
        action = model.decide(ctx.render())   # Output: call a tool or return a final answer
        # 2) Act: dispatch and execute the tool
        if action.kind == "answer":
            return action.text
        result = dispatch(action.tool, action.args, ctx)
        # 3) Observe: write the execution result back to the context for the next round
        ctx.append_tool_result(action, result)
    return "Maximum number of steps reached; human intervention is required"

Although this loop looks simple, a series of engineering mechanisms are required in practice:

1. Termination conditions: The loop must be able to end when the model outputs a final answer, when the task is completed, when the maximum number of steps is exceeded, or when the budget is exhausted.
2. Budget control: Set limits on the number of steps, elapsed time, and cost, such as token consumption. When a task is urgent, a “plan first, execute later” mode can be used: the model produces a complete list of steps at once, reducing the number of round trips.
3. Error handling: Distinguish between “retryable errors,” such as timeouts and rate limits, and “non-retryable errors,” such as invalid parameters. Retries should use exponential backoff to avoid turning failures into a storm.
4. State persistence: For long-running tasks, intermediate state should be written to local files to support resuming from breakpoints, human takeover, and later review.

3.2 Tool Registration: Letting the Model Know What It Can Do

Tools are the hands and feet through which agents connect to the external world. Each tool needs a clear description, strict parameter validation, a deterministic return structure, and a statement of idempotency. An example is shown below:

{
  "name": "search_repo",
  "description": "Search code snippets in a specified repository by keyword and return file paths, line numbers, and matched content",
  "parameters": {
    "type": "object",
    "properties": {
      "keyword": {
        "type": "string",
        "description": "Search keyword"
      },
      "path": {
        "type": "string",
        "description": "Repository root directory"
      }
    },
    "required": ["keyword"]
  },
  "returns": "A list of matches, including path, line number, and content snippet",
  "idempotent": true,
  "risk_level": "read_only",
  "timeout_sec": 30
}

There are three practical principles for designing tools:

• Appropriate granularity: If tools are too coarse, the model cannot control them precisely. If they are too fine-grained, too many steps are required to complete a task. A better approach is to provide a small number of “high-leverage” tools and allow them to be combined through scripts.
• The description is the prompt: The description of a tool directly determines whether the model can select it correctly. It should explain the purpose, applicable scenarios, and non-applicable scenarios, rather than just providing a name.
• Controlled returns: Files, logs, and search results can be very large. Truncation, summarization, or pagination should be used to prevent the context from being filled instantly.

4. Execution and Permissions: Let AI Do Work Without Causing Damage

4.1 Sandboxing: Defining the Boundaries of Capability

Tools run inside a sandbox. Common isolation methods include:

• Filesystem: Only mount the working directory, mark critical paths as read-only, and prohibit access to system directories.
• Command execution: Prefer an allowlist rather than a blocklist. Perform semantic parsing for combinations such as pipes, redirections, and wildcards to prevent bypass.
• Network access: Deny by default or allow only specified destinations. If network access is truly required, use a proxy and keep records.
• Resource limits: Set quotas for CPU, memory, and runtime to prevent infinite loops from exhausting machine resources.

sandbox:
  filesystem:
    roots: ["./workspace"]
    read_only_paths: ["./workspace/config", "/etc"]
  network:
    default: deny
    allow_domains: ["internal documentation service"]
  resources:
    cpu_limit: "2"
    memory_limit: "2GiB"
    max_runtime_sec: 120
  exec:
    mode: allowlist
    allowed_commands: ["git", "python", "grep", "cat"]

4.2 Risk-Based Permissions: The Higher the Risk, the Stronger the Constraints

Not all operations carry the same level of risk. Reading files is not on the same level as deleting data or deploying to production. Managing actions by risk level is a safer approach:

Level	Typical Operations	Execution Strategy
Read-only	View files, search code, query read-only data	Execute automatically and record logs
Local write	Create or modify files within the workspace	Execute automatically; changes must be traceable and can be rolled back
Mutating operations	Delete files, run installation commands, modify configurations	Execute only after human confirmation
High risk	Production operations, sending external data, paid operations	Disabled by default; must be explicitly enabled and require double confirmation

def guarded_exec(tool, args, ctx):
    level = tool.risk_level
    if level == "high_risk" and not ctx.policy.high_risk_enabled:
        return "Action rejected by policy: high-risk actions are not enabled"
    if level in ("mutating", "high_risk"):
        if not ctx.approve(tool.name, args):      # Show parameters and impact scope, then wait for human confirmation
            return "The user rejected the operation"
    result = sandbox.run(tool, args, timeout=tool.timeout_sec)
    ctx.audit_log.write(tool.name, args, result.status, operator=ctx.approve_by)
    return result

On this basis, an “abort” switch should also be retained: users can terminate a running task at any time and roll back the changes already made. For example, in a version control system, an independent branch can be created for the changes of a specific task to make comparison and rollback easier.

5. Memory and Context: Let Agents Remember and Find Information

The context window of a single conversation is limited, so long-running tasks inevitably face the problem of “too much information to fit.” Memory is usually divided into four types:

1. Working memory: The context window for the current task. It stores system instructions, user goals, and the reasoning and results of the most recent steps.
2. Episodic memory: Summaries and outcomes of historical tasks, such as “the last similar task failed because of insufficient permissions.”
3. Semantic memory: Knowledge introduced through retrieval, such as project documentation, coding standards, and historical incident cases. Keyword retrieval can be used first, followed by vector retrieval to balance exact matching and semantic similarity.
4. Procedural memory: Verified and effective operation patterns, such as standard handling steps for a certain type of task or reusable scripts. This is the “experience” accumulated by the agent.

When the context approaches the budget limit, compression is required:

If context length > budget threshold:
    Keep [system instructions, task goal, last N messages]
    Pass the remaining history to the model to generate a structured summary:
        - Completed: which files were modified and which commands were executed
        - Unresolved: error messages and possible causes
        - Next-step plan
    Replace the trimmed history with the summary

Writing episodic memory and procedural memory should also have a threshold. Only after a task succeeds and its results are verified can it be written into memory. Failure lessons should be annotated with the cause and applicable conditions. Otherwise, the memory store will be polluted by noise and mislead subsequent decisions.

6. Evaluation and Observability: Make Results Measurable and Problems Traceable

6.1 Task-Level Evaluation

“Being able to do work” must be measured by task completion, not by benchmark scores for the model itself. It is recommended to collect tasks from real scenarios and build a small evaluation set. Each task should have acceptance criteria that can be automatically judged:

• Programming scenarios: Whether all unit tests pass.
• Data processing scenarios: Whether the output matches the expected result.
• Operations scenarios: Whether the desired state is achieved without unauthorized operations during the process.

Metrics that need to be tracked include:

• Task success rate;
• Average number of steps and elapsed time;
• Human intervention rate;
• Cost per task;
• Number of unauthorized or dangerous operations.

Because model outputs are stochastic, the same task should be run multiple times to observe result stability instead of drawing conclusions from a single success.

6.2 Observability

Every tool call should output structured logs containing at least the task ID, step number, tool name, parameter summary with sensitive fields masked, result status, elapsed time, and cost, forming a complete replayable execution trace.

• Centralized storage: Logs should be consolidated into a logging service for easy search and long-term retention.
• Real-time monitoring: Use monitoring and alerting capabilities to set thresholds for sudden increases in failure rate, rising average steps, and increasing tool timeouts.
• Transport security: Task status and progress events can be asynchronously transmitted between local and cloud environments through a message queue to avoid long-running tasks blocking interaction.

One task corresponds to one trace, which is key to identifying “why it did something wrong.” Common problems are often not caused by the model itself, but by abnormal tool returns, contaminated context, or improper permission configuration.

7. Common Mistakes and Practical Recommendations

1. Designing multi-agent systems from the very beginning: Collaboration can bring division of labor, but it also brings coordination costs and error propagation. It is better to first stabilize the single-agent loop and then split agents according to task complexity.
2. Pursuing full automation: In mutating operations, human-in-the-loop is not a defect; it is a quality and safety mechanism. An agent can first run in “suggestion mode,” then gradually gain more permissions after trust is accumulated.
3. Ignoring tool return quality: The model can only see what tools return. Empty values, vague error messages, or overly long content can make the agent “get lost.”
4. Tuning prompts without evaluation: Adjusting prompts based on intuition easily leads to repeated back-and-forth. An evaluation set should be established first, and optimization should be driven by data.

For enterprise users, it is recommended to proceed in three stages: pilot, expansion, and platformization. First, validate a closed loop in one or two high-frequency, low-risk scenarios. Then gradually cover more processes and improve the permission system. Finally, consolidate the system into a unified platform that supports multiple users and multiple projects and shares tools and knowledge bases. During the expansion stage, teams can combine cloud capabilities such as model services, object storage, message queues, container platforms, logging services, and monitoring and alerting, allowing teams to focus on orchestration logic and scenario adaptation instead of rebuilding infrastructure repeatedly.

Conclusion

“Being able to actually do work” is not a feature of a single model. It is an engineering capability formed by models, tools, loops, permissions, and evaluation working together. Local-first provides a foundation of controllable data and trustworthy execution for this capability. As long as developers focus on the following six key points — loop control, tool design, sandbox isolation, risk-based permissions, memory management, and observability — they can build stable, secure, and evolvable agent systems, making AI a true and effective “digital colleague.”