Building Free AI Sidebars and Intelligent Copilots: Technical Practices

1. Problem Definition: Core Goals of Sidebar-based AI Assistants

A sidebar-based AI assistant provides a low-distraction, easily accessible auxiliary entry point within a user’s workspace. Rather than acting as an independent chat window that disrupts the main workflow, it leverages the current page, selected text, conversation history, and user preferences to offer features like explanation, summarization, rewriting, retrieval, task decomposition, and tool invocation. Its technical objectives can be summarized in four points: lightweight entry, accurate context, controllable response, and safe execution.

The requirement for “free basic services with limited free quotas for advanced models” is essentially a productization strategy: using free basic capabilities to lower the barrier to entry, while using limited advanced model quotas to support high-value scenarios. Costs are controlled through quotas, rate limiting, graceful degradation, and caching. From an engineering perspective, “free” does not mean unlimited API calls; it must be designed as a closed-loop system comprising tiered model routing, quota accounts, request governance, and observability.

2. Overall Architecture: Layered Decoupling and Governance

A six-layer architecture is recommended:

1. Entry Layer: Sidebar components, shortcut keys, selection awareness, page summary entry points, notifications, and status feedback.
2. Session Orchestration Layer: Session state machine, context assembly, prompt templates, streaming output, and cancellation mechanisms.
3. Model Routing Layer: Routing, degradation, retry, and timeout control between basic free models and limited advanced models.
4. Retrieval & Tool Layer: Knowledge base retrieval, page content indexing, tool registration, permission validation, and execution auditing.
5. Governance Layer: Quota accounts, rate limiting, caching, sensitive data masking, content security, and user authorization.
6. Infrastructure Layer: Container platforms, object storage, logging services, monitoring and alerting, and configuration management.

The key value of this layered approach is that the entry layer does not directly depend on model details, the model routing layer is unaware of page structures, and the governance layer processes requests and responses as a sidecar. This allows the system to support free basic services and smoothly degrade when advanced model quotas are exhausted.

3. Free Quota and Cost Governance: From “Giveaway” to “Controllable”

Productizing free quotas requires account-based, metered, and policy-driven approaches. Each user can be assigned basic and advanced quotas: basic quotas handle low-risk, short-context tasks, while advanced quotas handle long-document summarization, complex reasoning, or multi-tool orchestration. The routing strategy should prioritize “task completion first, cost control second”:

function routeModel(request, user):
    quota = quotaService.query(user.id)
    task = classifyTask(request.input, request.pageContext)
    if task.complexity == "high" and quota.advancedRemaining > 0:
        model = "advanced-limited"
    else if task.complexity == "medium" and cache.has(request.fingerprint):
        model = "basic-free"
    else:
        model = "basic-free"
    if quota.totalRemaining <= 0:
        return degradedResponse("Quota exhausted, switching to basic capabilities")
    return modelGateway.stream(model, request)

Beyond model routing, multiple cost control methods should be combined: caching results for high-frequency similar requests; summarizing and chunking long contexts for retrieval; queuing and batching non-real-time tasks; applying bounded retries for failed requests; and setting per-user, per-session, and global concurrency thresholds for advanced model calls. Quota notifications should be clear without aggressively pushing upgrades, avoiding designs that force users to upgrade to access free features.

4. Sidebar Interaction: Low Distraction and Cancellable

Sidebar interactions should follow the principles of “non-blocking, cancellable, and traceable.” When a user selects text, the sidebar should display suggested actions rather than executing automatically. Streaming output should allow users to stop at any time, long tasks should display phase statuses, and failures should provide recoverable suggestions. The frontend state machine can be abstracted into: idle, requesting, streaming, waiting for tool confirmation, completed, failed, and cancelled. Each state must have clear user feedback and timeout handling.

Streaming can use general event streams or long-polling protocols, but requires first-byte timeouts, idle timeouts, and maximum output length limits. For page content reading, the system should prioritize explicitly selected fragments, followed by summarized page structures, to avoid the privacy and cost risks of uploading the entire page at once.

5. Context Management and Retrieval-Augmented Generation

Context should be assembled in tiers: the user’s current selection, recent messages in the current session, user preferences, structured page summaries, and permission-filtered knowledge base snippets. Long contexts should not be directly concatenated; instead, they require denoising, chunking, summarization, and relevance ranking. The RAG workflow includes: query rewriting, vector or keyword retrieval, permission filtering, reranking, citation concatenation, and source tracing.

function buildContext(request, user):
    selection = sanitize(request.selection)
    session = compressSession(request.session, maxTokens=800)
    pageSummary = summarizePage(request.pageStructure)
    docs = retrieve(request.input, filters=user.permissions)
    return composeTemplate({
        selection, session, pageSummary, docs,
        citationRequired: true
    })

Citation tracing reduces hallucination risks and makes it easier for users to verify information. For knowledge base content, permission filtering must be completed during the retrieval phase, not patched after generation.

6. Tool Calling and Automated Execution

The value of an intelligent copilot extends beyond Q&A to include tool invocation. Tools should declare their name, purpose, input structure, permission scope, timeout, and idempotency requirements through a unified registry. High-risk operations require secondary confirmation, and execution processes must log audit trails. Tool calls should have maximum step limits, timeouts, retry ceilings, and sandbox isolation to prevent infinite loops or unauthorized executions.

7. Security, Privacy, and Compliance

Security design should cover least privilege, transport encryption, storage isolation, log masking, prompt injection defense, and output filtering. Users should be able to view and clear session data, and to disable page awareness capabilities. Sensitive fields should be masked locally or server-side; model outputs must be filtered for sensitive information and unsafe content; and tool executions require strict permission validation and approval.

8. Observability and Quality Evaluation

It is recommended to monitor request volume, first-byte latency, end-to-end latency, success rate, quota consumption, cache hit rate, degradation rate, tool call success rate, and user feedback. Establish regression test sets covering summarization, rewriting, retrieval Q&A, and tool orchestration scenarios. Use canary releases to compare the impact of different routing strategies on cost and user satisfaction. Alerts should differentiate between entry errors, model errors, retrieval errors, and quota anomalies.

9. Deployment and Engineering Practices

For infrastructure, backend services for the sidebar assistant can be built using general-purpose services such as container platforms, object storage, logging services, and monitoring/alerting provided by Baidu AI Cloud. Deployment should separate development, staging, and production environments. A configuration center should manage model routing and quota policies, while canary releases control the impact scope of new strategies. Capacity planning should estimate based on concurrent session counts, average context length, and advanced model quota consumption.

10. Implementation Checklist

1. Define the boundary between free basic capabilities and limited advanced quotas.
2. Establish user quota accounts and the metering pipeline.
3. Implement model routing, degradation, caching, and rate limiting.
4. Design the sidebar state machine and cancellable streaming output.
5. Complete context compression, RAG, and citation tracing.
6. Establish tool registry, permission approval, and auditing.
7. Implement data masking, content filtering, and user authorization.
8. Build monitoring alerts, regression tests, and canary release mechanisms.

In summary, the engineering focus of free AI sidebars and intelligent copilots is not merely connecting to a model, but building a complete technical loop around low cost, controllability, security, and observability. Only by integrating quota governance, context management, and secure tool execution into a unified architecture can a stable balance be achieved between free basic services and premium capabilities.