Building AI Agents with LLM APIs: A Guide to Technical Architecture and Engineering Practices

    Large language model (LLM) open platforms—commonly delivered as Model-as-a-Service (MaaS)—expose standardized remote inference through HTTP APIs. The real engineering challenge is not calling an endpoint but building reliable, controllable, and continuously evolvable systems around model capabilities. This guide follows the full path—from architectural understanding to integration design, advanced capabilities, governance, and a phased rollout. It covers authentication and key management, request design and parameter tuning, error handling with backoff, streaming output, structured output, concurrency and rate limiting, multimodal understanding, agent and tool orchestration, retrieval-augmented generation (RAG), cost and performance governance, and security and compliance. Practical Python examples are included for retries, structured-output validation, and an agent execution loop, along with engineering guidance for teams that want to turn model APIs into dependable business applications.

    1. Understanding the Open Platform: From “Calling a Model” to “Building a System”

    Large model open platforms are often described as Model-as-a-Service (MaaS). They wrap foundation models as standardized remote inference services, so developers can access text generation, multimodal understanding, and agent building over HTTP interfaces—without procuring GPU clusters or deploying and maintaining inference frameworks.

    For engineering teams, the real challenge is not “whether you can call an API” but how to build a reliable, controllable, and continuously evolvable business system around model capabilities. That pipeline involves at least the following stages:

    • Authentication and key management;
    • Request design and parameter tuning;
    • Error handling and retry backoff;
    • Streaming output;
    • Multimodal input handling;
    • Agent orchestration;
    • Knowledge base augmentation;
    • Cost and performance governance;
    • Observability and security compliance.

    This article walks through them in order: architectural understanding → integration design → advanced capabilities → governance and optimization → rollout path.

    2. Layered Architecture: What Makes Up an Open Platform

    From a systems perspective, an open platform can be abstracted into five layers:

    Layer Main responsibility Key content
    Model layer Provides foundation models of different sizes and modalities General chat, long context, reasoning-enhanced, multimodal; often a flagship version and a lightweight high-speed version
    Capability layer Packages model capabilities into composable business capabilities Text generation, multimodal understanding, embedding and retrieval, tool calling, agent orchestration
    API gateway Unified entry point handling authentication, rate limiting, and routing API key authentication, quota management, streaming, model version routing
    Metering & governance Usage statistics, billing, and content safety Token metering, usage dashboards, quota control, content moderation
    Infrastructure Supports the operation of business systems Object storage, message queues, container platform, logging, monitoring and alerting, and other general cloud services

    The model layer sets the capability boundary. Different models make different trade-offs in context length, multimodal support, output speed, and unit cost. In practice, models are usually not substitutes for one another but divide the labor:

    • Route complex reasoning and long-form writing to stronger flagship models;
    • Route high-frequency tasks like classification, extraction, and rewriting to lightweight, high-speed versions.

    The capability layer is the source of business value. A single “generate text from one prompt” call rarely solves a real business problem; value comes mostly from combining capabilities. Enterprise knowledge base Q&A, for example, typically chains together document parsing → chunking and embedding → hybrid retrieval → reranking → generation with citations.

    The gateway layer determines stability. For the integrating side, the gateway is the actual call entry point. Engineering practices like key management, retry strategies, and concurrency control all revolve around this layer.

    The metering and governance layer determines controllability. Public research analyses show that token consumption often concentrates in a small number of long-context, high-output calls. Usage statistics and quota management should span the entire lifecycle of development, testing, and production.

    The infrastructure layer determines sustainability. Business systems typically rely on general cloud services:

    • Object storage: holds raw data such as documents, images, and audio/video;
    • Message queues: asynchronously decouple and smooth out time-consuming tasks like document parsing and batch generation;
    • Container platform: hosts business services and scales elastically with load;
    • Logging: centrally collects call records to support troubleshooting and auditing;
    • Monitoring and alerting: continuously observes success rate, latency, rate limiting, and cost.

    3. Integration Design: Six Keys to Stable Calls

    3.1 Authentication: Keys Are Infrastructure, Not a Config File

    Calling an inference service usually relies on API key authentication, with the credential carried in the request header. Engineering practices to follow:

    • Environment isolation: use different keys for development, testing, and production to simplify traceability and revocation;
    • Least privilege: grant only the model scope and quotas each business needs;
    • Centralized custody: store keys through a key management service, and never hardcode them in code repositories, images, or front-end pages;
    • Regular rotation: establish a rotation schedule and an emergency plan, validating a new key before revoking the old one;
    • Call tracing: log key identifiers, but never log key plaintext.

    3.2 Request Design: Every Parameter Has a Clear Purpose

    A typical chat completion request usually contains three kinds of content:

    • Model identifier: determines the capability boundary, context length, and billing tier;
    • Message list: roles generally include system (setting the role, constraints, and output format), user (the user’s task), and assistant (the model’s prior replies); multi-turn conversations are appended in chronological order;
    • Generation parameters: control randomness, output length, and how results are returned. Common parameters are shown below.
    Parameter Role Recommendation
    temperature Controls sampling randomness, usually 0–2 Use low values for deterministic tasks like classification and extraction; raise it for creative generation
    top_p Truncates candidate tokens by cumulative probability Usually tune only one of temperature or top_p, not both, to avoid unpredictable combined effects
    max_tokens Caps the number of output tokens Must cover the expected output length so replies are not truncated mid-way
    stop Custom stop sequences Good for scenarios that output in a fixed format
    stream Whether to return a stream Recommended for end-user-facing scenarios

    Prompts are best structured in layers: role—task—constraints—output format—examples, where:

    • the role defines the model’s perspective and tone;
    • the task describes the problem to solve;
    • the constraints clarify boundaries, such as no fabrication, the data cutoff range, and prohibited actions;
    • the output format and examples stabilize the result structure so downstream programs can parse it.

    Example (System message):

    1. Role: You are the after-sales support assistant for an e-commerce platform.
    2. Task: Based on the [order information] and [refund policy], determine the
    3. user's request and provide a handling recommendation.
    4. Constraints:
    5. 1. Answer only from the provided material; do not infer anything missing from it.
    6. 2. When money amounts are involved, show the calculation basis.
    7. 3. If the material is insufficient to decide, output "human intervention required" and explain why.
    8. Output format: Strictly output JSON with the fields conclusion, reason, amount, need_human.

    3.3 Error Handling: Retrying Is Not “Fail and Just Start Over”

    Calling a remote inference service will inevitably encounter failures. The key is to distinguish error types and apply different strategies:

    • Retryable: network timeouts, connection drops, 5xx server errors, 429 rate limiting;
    • Non-retryable: 400 parameter errors, 401/403 authentication failures, a nonexistent model identifier, context exceeding the model’s limit.

    For retryable errors, use exponential backoff with random jitter to avoid many clients retrying at the same instant and forming a “retry storm.” If the response includes a suggested retry wait, defer to it. Also set a maximum retry count and an overall timeout so requests never hang indefinitely.

    1. import json
    2. import time
    3. import random
    4. import urllib.request
    5. BASE_URL = "<inference service URL>"
    6. HEADERS = {"Content-Type": "application/json", "Authorization": "Bearer <API_KEY>"}
    7. RETRYABLE_STATUS = {429, 500, 502, 503, 504}
    8. class ApiError(Exception):
    9. def __init__(self, status, message, retryable):
    10. super().__init__(f"HTTP {status}: {message}")
    11. self.status = status
    12. self.retryable = retryable
    13. def call_api(path, payload, max_retries=3, timeout=30):
    14. data = json.dumps(payload).encode("utf-8")
    15. last_err = None
    16. for attempt in range(max_retries + 1):
    17. try:
    18. req = urllib.request.Request(
    19. f"{BASE_URL}{path}", data=data, headers=HEADERS, method="POST"
    20. )
    21. with urllib.request.urlopen(req, timeout=timeout) as resp:
    22. return json.loads(resp.read().decode("utf-8"))
    23. except urllib.error.HTTPError as e:
    24. body = e.read().decode("utf-8", errors="ignore")[:500]
    25. err = ApiError(e.code, body, e.code in RETRYABLE_STATUS)
    26. except (urllib.error.URLError, TimeoutError) as e:
    27. # Connection failure or read timeout — treat as retryable
    28. err = ApiError(0, str(e), True)
    29. last_err = err
    30. if not err.retryable or attempt == max_retries:
    31. raise err
    32. delay = min(2 ** attempt, 8) + random.uniform(0, 1) # exponential backoff + jitter
    33. time.sleep(delay)
    34. raise last_err

    Whatever HTTP client you use, retry design should follow three principles:

    1. Be cautious with write operations: for write actions like billing or placing orders, model inference itself has no side effects, but the subsequent business execution must be idempotent to avoid running twice;
    2. Bound the retries: set a maximum retry count and an overall timeout, then fall back to a degraded or manual path once the limit is exceeded;
    3. Isolate failures: give retried requests their own timeout budget so a few slow requests cannot drag down the entire call chain.

    3.4 Streaming Output: Making the “Wait” Perceptible

    Generative models emit output token by token, and long answers can take seconds or even tens of seconds. By enabling streaming, the platform typically pushes text incrementally over Server-Sent Events (SSE), so callers can receive and render as they go.

    Things to watch with streaming:

    • Parse increments: parse each chunk, concatenate the incremental content, then render, to avoid duplicating or dropping text;
    • Detect the end event: distinguish normal completion, hitting the output length limit, and abnormal interruption;
    • Handle connection drops: already-displayed content may be incomplete, so prompt the user to retry;
    • Coordinate front end and back end: when the backend proxies the stream, disable unnecessary buffering so data is delivered promptly.

    3.5 Structured Output: Making Model Results Consumable by Programs

    Business systems usually need structured results. There are three common approaches:

    • Prompt constraints: explicitly request JSON and describe each field’s meaning;
    • Schema constraints: some platforms support defining the output structure with formats like JSON Schema, letting the server enforce the model’s output;
    • Tool calling: obtain structured results indirectly through a function’s parameter structure.

    Whatever the approach, validate again on the client side:

    1. import json
    2. from jsonschema import validate, ValidationError
    3. ORDER_SCHEMA = {
    4. "type": "object",
    5. "properties": {
    6. "conclusion": {"type": "string"},
    7. "reason": {"type": "string"},
    8. "amount": {"type": "number"},
    9. "need_human": {"type": "boolean"},
    10. },
    11. "required": ["conclusion", "reason", "amount", "need_human"],
    12. }
    13. def parse_structured(text):
    14. # Fallback 1: extract the first complete JSON object in the text
    15. start, depth, end = text.find("{"), 0, -1
    16. if start == -1:
    17. raise ValueError("No JSON found in the response")
    18. for i in range(start, len(text)):
    19. if text[i] == "{":
    20. depth += 1
    21. elif text[i] == "}":
    22. depth -= 1
    23. if depth == 0:
    24. end = i + 1
    25. break
    26. if end == -1:
    27. raise ValueError("Incomplete JSON")
    28. # Fallback 2: validate against the schema so the result can be consumed directly by business code
    29. result = json.loads(text[start:end])
    30. validate(instance=result, schema=ORDER_SCHEMA)
    31. return result

    3.6 Concurrency and Rate Limiting: The Client Needs “Brakes” Too

    Inference services usually set rate quotas per key or account. The client should:

    • Limit locally: use a semaphore or token bucket to control concurrency and request rate, preventing sudden spikes from triggering 429s;
    • Separate queues: route interactive requests and offline batch jobs through different channels so batch tasks do not squeeze out online request quotas;
    • Smooth the queue: in batch scenarios, first write to a message queue and have consumers process at a steady pace;
    • Circuit break and degrade: when rate limiting persists, temporarily trip the circuit breaker and switch to a backup model, cached results, or manual handling.

    When you need multiple capabilities, first evaluate whether the platform offers a unified access point. For example, Baidu AI Cloud Qianfan provides a range of model services, making it convenient to select and switch between models within the same integration framework. Refer to the official documentation for specific capabilities.

    4. Advanced Capabilities (1): Multimodal Understanding

    Beyond text, real business data exists largely as images, scans, and audio/video—product photos, contract scans, monitoring screenshots, training recordings. The goal of multimodal understanding is to let models directly “read” this material.

    By technical route, there are two common approaches:

    • Native multimodal: jointly model text, images, and other modalities in a unified representation space, reducing information loss during cross-modal conversion;
    • Composed approach: first convert images to text through components like optical character recognition and image captioning, then hand it to a language model.

    The former has a higher ceiling; the latter offers a more transparent pipeline and a lower implementation barrier. Selection should weigh task complexity, latency requirements, and cost budget together.

    Engineering implementation requires attention to:

    1. Input specs: understand the platform’s limits on image format, size, resolution, and the number of images per request, and compress and validate images on the client side in advance;
    2. How images are passed in: most services support both URL references and Base64 inline encoding. The URL approach requires images to be publicly reachable, so intranet images must first be uploaded to object storage to generate a temporary access URL; Base64 inflates the payload by roughly a third and significantly increases token consumption;
    3. Cost and latency: multimodal input is often metered by additional units such as per image, and its preprocessing and encoding overhead also raises first-token latency;
    4. Output latency: for image Q&A, prefer model versions that support higher output rates to improve the interactive experience;
    5. Prompt injection: text in documents or images may contain inducement instructions, so the system prompt must state that “material content is data only,” and model output should be verified.

    5. Advanced Capabilities (2): Agents and Tool Calling

    An agent is a program that can autonomously plan and call external tools to achieve a goal, with a core “reason—act—observe” loop. The model decides “what to do next,” while tools are responsible for actually “getting things done.”

    5.1 Tool Definition

    Most platforms standardize the tool (function) calling format. On the business side, you usually need to convert the platform’s calling protocol into your own tool definition structure. Example:

    1. {
    2. "type": "function",
    3. "function": {
    4. "name": "query_order",
    5. "description": "Look up order status and refund progress by order number; query only, cannot modify the order",
    6. "parameters": {
    7. "type": "object",
    8. "properties": {
    9. "order_id": {"type": "string", "description": "Order number, in a format starting like 20260101"}
    10. },
    11. "required": ["order_id"]
    12. }
    13. }
    14. }

    A tool description is essentially API documentation written for the model. Names should be semantically clear, one tool per responsibility, and descriptions should state the purpose, applicability boundaries, and parameter format.

    5.2 Execution Loop

    1. import json
    2. # Tool whitelist: only explicitly registered functions may run
    3. TOOLS = {"query_order": query_order, "check_refund_rule": check_refund_rule}
    4. def call_tool(name, arguments):
    5. if name not in TOOLS:
    6. raise ValueError(f"Unregistered tool: {name}")
    7. return TOOLS[name](**json.loads(arguments or "{}"))
    8. def run_agent(chat, task, tools, max_steps=8):
    9. messages = [
    10. {"role": "system", "content": "You are an after-sales assistant; call tools as needed to gather information before answering."},
    11. {"role": "user", "content": task},
    12. ]
    13. for _ in range(max_steps):
    14. resp = chat(messages, tools=tools)
    15. message = resp["message"]
    16. messages.append(message)
    17. tool_calls = message.get("tool_calls") or []
    18. if not tool_calls:
    19. return message.get("content", "") # the model produced a final answer
    20. for call in tool_calls:
    21. func = call["function"]
    22. try:
    23. result = call_tool(func["name"], func.get("arguments"))
    24. observation = {"ok": True, "data": result}
    25. except Exception as e:
    26. # Feed the tool failure back to the model and let it decide whether to retry or take another path
    27. observation = {"ok": False, "error": str(e)}
    28. messages.append({"role": "tool", "content": json.dumps(observation, ensure_ascii=False)})
    29. return "Task exceeded the step limit; handed off to a human"

    5.3 Key Design Principles

    • Step and budget limits: prevent the model from looping on tool calls;
    • Read/write separation: query tools may run automatically, while write operations such as placing orders, refunds, and deletions must require human confirmation or approval;
    • Result validation: validate both tool return values and the model’s final output separately—for example, check amount calculations and format compliance;
    • Conversation memory: when the context window is limited, summarize earlier dialogue or store history in external storage and dynamically trim it against a token budget;
    • Multi-agent collaboration: complex flows can be split into roles such as planner, executor, and reviewer, passing tasks and results through a message queue.

    6. Knowledge Base Augmentation: Engineering Techniques to Mitigate Hallucination

    A model’s knowledge comes from its training data and cannot cover private information like internal company policies and product manuals, so it tends to produce answers that “seem plausible but are factually wrong.” Knowledge base augmentation is commonly called retrieval-augmented generation (RAG), with the following basic flow:

    1. Data ingestion: store documents, web pages, database records, and so on uniformly in object storage or a data service;
    2. Parsing and cleaning: extract body text by document type, remove noise like headers, footers, and watermarks, and preserve row/column structure for tables as much as possible;
    3. Chunking and embedding: split into semantically complete chunks along semantic boundaries with appropriate overlap, then call the platform’s embedding API to generate vectors;
    4. Index storage: write vectors to a vector database and original text to a relational or document database, linking vectors and original text both ways;
    5. Hybrid retrieval: vector retrieval handles semantic similarity, keyword retrieval handles exact matches for proper nouns and IDs, and the two are fused for recall;
    6. Reranking: use a reranking model to finely rank candidate chunks, putting only the few most relevant ones into the context;
    7. Generation and citations: the model answers based on the chunks and cites its sources.

    Common problems in practice and how to address them:

    • Chunking breaks semantics: prefer splitting along structural boundaries like headings and paragraphs, and avoid cutting in the middle of tables or lists;
    • Nothing is retrieved: expand synonyms and business terms, and build a Q&A-pair index for high-frequency questions;
    • Off-topic answers: appropriately relax the recall count and use reranking to control noise;
    • Fabrication: emphasize in the prompt “state clearly when the material is insufficient,” and programmatically verify numbers, dates, and clauses in answers.

    For evaluation, build a fixed Q&A test set and separately measure retrieval hit rate, answer accuracy, and citation accuracy as a baseline for each round of adjustments.

    7. Cost and Performance Governance

    7.1 Metering Dimensions and Influencing Factors

    The unit of metering for inference services is usually the token, the basic unit of text the model processes. The token count for Chinese, English, code, and images is not the same. Token consumption can be broken down as:

    1. Total tokens = input tokens + output tokens
    2. Input tokens = system prompt + conversation history + retrieved chunks + user input

    As you can see, long conversations, multi-turn tool calls, and large retrieved content all drive up cost. Multimodal requests may add extra metering units such as per image. Most platforms price input and output separately, with output usually costing more.

    7.2 Eight Optimization Tactics

    1. Model tiering: first use a lightweight model for intent recognition or pre-classification, and route only complex requests to a flagship model;
    2. Trim prompts: remove redundant examples and extract stable system prompts so they can hit context-caching features;
    3. Compress context: summarize conversation history, deduplicate retrieved chunks, and limit their count;
    4. Cap output length: set a reasonable output limit and ask the model to “lead with the conclusion” to cut redundant padding;
    5. Cache results: cache results for high-frequency repeated requests, set an expiration, and avoid personalized information;
    6. Batch processing: merge requests for offline tasks and use batch endpoints where the platform supports it;
    7. Usage monitoring: measure token consumption by business line, environment, and key, and set budget alerts;
    8. Periodic evaluation: before shipping any optimization, validate quality on the test set to avoid “saving cost while losing quality.”

    7.3 Performance and Stability

    Generative services are usually measured by two metrics:

    • First-token latency: the time from sending the request to receiving the first output token;
    • Output rate: the average generation speed of subsequent tokens, usually expressed in tokens/second.

    Together they determine the perceived responsiveness. The main engineering tactics include:

    • streaming output to reduce perceived waiting;
    • controlling input length to cut prefill-stage overhead;
    • capacity planning based on the model specs and quotas the platform provides.

    For stability, build a monitoring dashboard covering these metrics: request volume, success rate, error code distribution, rate-limit count, first-token latency, output rate, token consumption, and cost. Also set tiered alerts on error rate and latency, and periodically run scale-up drills on the container platform to verify peak-load capacity.

    8. Security, Compliance, and Governance

    • Data classification: handle public, internal, and sensitive data separately, and de-identify or redact sensitive information before sending it to the model;
    • Transport and storage security: use encrypted transport end to end, centralize key custody, and prohibit logging sensitive raw text;
    • Two-way content safety: moderate both input and output to block illegal or harmful information and defend against prompt injection;
    • Permissions and auditing: grant model and knowledge base access on the least-privilege principle, and fully record calls, tool executions, and human approvals for traceability;
    • Compliance assessment: before offering a service to the public, confirm you have met applicable regulatory obligations such as any required registrations and security assessments, and keep track of regulatory updates;
    • Vendor resilience: design primary/backup model switching and fallback plans for critical business so a single model’s outage does not interrupt service.

    9. Phased Rollout Roadmap

    Phase Core tasks Key outcomes
    Trial validation Set up authentication; validate prompts and output quality A go/no-go conclusion on feasibility; a prompt baseline
    Engineering Retry backoff, streaming output, structured parsing, logging and monitoring A stable, callable service wrapper
    Capability expansion Integrate knowledge base, multimodality, and tool calling A business-ready intelligent application
    Scale operations Model tiering, caching, quotas, evaluation, and alerting A cost-controlled, quality-measurable operating system

    Conclusion

    Open platforms lower the technical barrier to using large models, but they do not lower engineering complexity. Every stage—authentication, retries, streaming, structuring, multimodality, and agent orchestration—directly affects an application’s stability, cost, and user experience. Only by treating the model as “one component” of the system and building an observable, degradable, and measurable engineering foundation around it can enterprises truly unlock the business value of large model technology.