Connecting Large Language Models to Data: Combining RAG and Agentic AI in Practice

Large language models (LLMs) are strong at understanding, summarizing, and generating language. However, their parametric knowledge has a training cutoff, limited domain coverage, and no built-in traceability. When a model must answer questions about private enterprise knowledge, live business status, or data spread across several systems, prompting alone rarely produces reliable results. Many tasks also go beyond a single answer. They require breaking down a goal, retrieving evidence, calling tools, verifying results, and revising the plan. Retrieval-Augmented Generation (RAG) and agentic AI are two complementary approaches. RAG determines what context the model receives. Agentic AI determines how the model acts toward a goal. Combined, they can turn an LLM from a one-off question-answering tool into the core of an application that connects to data, runs workflows, and supports governance and evaluation.

1. Problem Background: Why LLMs Need Data Connections

An LLM’s knowledge comes from its training data. This makes it suitable for general language tasks, but enterprise scenarios usually expose three gaps.

The first is the factual gap. Internal policies, product documentation, support tickets, and business metrics are not stored in the model’s parameters, so the model cannot know them by default.

The second is the freshness gap. Prices, inventory, order status, and approval progress change constantly. They must be retrieved in real time or near real time.

The third is the trust gap. A model may generate content that looks plausible but has no supporting evidence. Without citations, validation, and access control, such output is difficult to deploy in production.

“Connecting data” therefore does not mean pasting documents into a prompt. It means building a governed data pipeline. Data can be ingested, indexed, retrieved, reranked, cited, and monitored. The system returns content according to the user’s permissions. RAG provides the basic form of this pipeline. Agents add planning, tool calls, and closed-loop verification on top of it.

2. RAG: Turning External Data into Controlled Context

The core idea of RAG is to retrieve first and then generate. Based on the user’s question, the system retrieves relevant passages from external knowledge sources. It passes these passages to the LLM as context and requires the model to answer based on that evidence.

A complete RAG pipeline usually includes data ingestion, preprocessing, indexing, retrieval, reranking, context assembly, generation, and citation.

1. Data Ingestion and Preprocessing

Data sources can include documents, web pages, database records, tickets, logs, and structured reports. In practice, the pipeline must handle incremental synchronization, version management, removal of outdated content, permission labels, and data quality cleaning.

For unstructured documents, the chunking strategy directly affects retrieval quality. Chunks that are too small lose meaning. Chunks that are too large add noise. Teams can chunk by heading hierarchy, semantic paragraphs, or a combination of parent and child chunks. Each chunk should carry metadata such as source, update time, department, and security classification. This metadata supports later filtering and citation.

2. Indexing and Hybrid Retrieval

Vector search handles semantic similarity well, but it may miss proper nouns, identifiers, and exact matches. Keyword search finds exact matches well, but its ability to generalize is limited. A more reliable approach is hybrid retrieval. The system retrieves candidates from both a vector index and a keyword index, then narrows the set with metadata filters.

For example, when answering a policy question, the system can first filter by the user’s department and document classification, then run semantic retrieval. This reduces the risk of unauthorized access and irrelevant results.

3. Reranking and Context Assembly

The initial retrieval often returns many results. A reranking model or rule-based strategy then selects the most relevant passages.

Context assembly must respect the token budget. Place the most relevant, most recent, and most authoritative content first. Compress or summarize long documents when needed.

The prompt should state clear rules: answer only from the provided evidence, say so when the evidence is insufficient, and cite the sources. These rules reduce hallucinations and make answers traceable.

4. Generation, Citation, and Refusal

Generation is not simply concatenating an answer. The system organizes the question, evidence, output format, and citation rules together.

For high-risk scenarios, the system should support refusal and escalation to a human. Citations can point to the document title, passage identifier, and update time, so users can verify the answer.

The engineering goal of RAG is not to make the model know everything. The goal is to let the model answer reliably within the boundaries of the evidence it is given.

3. Agentic AI: From Single Answers to Closed-Loop Actions

Agentic AI emphasizes goal-driven behavior and closed-loop action. An agent typically includes modules for goal interpretation, task planning, tool selection, memory management, result verification, and termination decisions.

Compared with simple question answering, an agent must decide what to do next. It may search the knowledge base, query a database, call a business interface, or ask the user for more information.

Tool calling is a key capability of agents, and it is also a source of risk. Production systems should:

• Maintain an allowlist of tools and validate parameter schemas.
• Set timeouts, retries, and idempotency, and keep audit logs.
• Require human confirmation or secondary authorization for write operations.
• Inject user identity and permission context in multi-tenant scenarios.

Memory can be divided into short-term task memory and long-term preference memory. Short-term memory stores evidence and intermediate results for the current task. Long-term memory should be written carefully and must support expiration and deletion.

Reflection checks whether an answer matches its evidence and whether tool results look abnormal. When a step fails, reflection can adjust the plan or roll back.

4. Combining RAG and Agents: Agentic RAG

Placing RAG inside an agent loop creates Agentic RAG, a system that can plan, retrieve, verify, and act. Typical patterns include:

1. Query planning and multi-hop retrieval. A complex question is split into sub-questions. Each sub-question is retrieved separately, and the results are then combined. For example, the question “Compare the feature differences between two versions and recommend a migration path” requires first retrieving the version release notes, then the migration guide, and finally composing the answer.

2. Data source routing. The agent chooses a knowledge base, a structured query interface, or a real-time business tool based on the question type. Routing can rely on metadata, intent classification, or rules.

3. Tool-augmented retrieval. When the knowledge base lacks real-time status, the agent calls a query interface. The returned result then serves as evidence for generation.

4. Verification and reflection. Before generation, the agent checks whether the evidence covers the question. After generation, it checks whether citations match the conclusions. If the evidence is insufficient, the agent retrieves again or refuses to answer.

5. Budget and termination control. Set limits on steps, tool calls, time, and cost to prevent loops and wasted resources.

An Agentic RAG system must still prioritize evidence. The agent can plan actions, but it cannot bypass permissions, audit requirements, or citation rules. Tools can extend capabilities, but they cannot replace a verifiable data pipeline.

5. Reference Architecture and Pseudocode

A practical architecture can be divided into six layers: data, indexing and retrieval, orchestration, tools, generation, and governance and evaluation.

• The data layer handles ingestion and cleaning.
• The indexing and retrieval layer provides hybrid search, filtering, and reranking.
• The orchestration layer manages the RAG workflow and the agent loop.
• The tool layer wraps controlled interfaces.
• The generation layer handles prompts, citations, and refusals.
• The governance layer manages permissions, masking, logging, monitoring, and evaluation.

On general-purpose cloud platforms such as Baidu AI Cloud, these layers can be built from common services, including object storage, message queues, container platforms, log services, and monitoring and alerting. The core design, however, remains the data pipeline and the governance rules.

def rag_answer(user_query, user_context):
    query = rewrite_query(user_query, history=[])
    filters = build_permission_filter(user_context)
    query_vector = embed([query])[0]
    candidates = hybrid_search(
        query=query,
        vector=query_vector,
        top_k=20,
        filters=filters
    )
    evidences = rerank(query, candidates, top_n=6)
    context = assemble_context(evidences, max_tokens=3000)
    prompt = build_prompt(
        question=query,
        context=context,
        rules="Answer only based on the evidence; provide citations; refuse to answer if the evidence is insufficient"
    )
    answer = llm_generate(prompt)
    return validate_and_cite(answer, evidences)
def agent_run(goal, user_context):
    memory = []
    for step in range(MAX_STEPS):
        plan = llm_plan(goal, memory)
        if plan.action == "retrieve":
            docs = retrieve(plan.query, user_context)
            memory.append(("evidence", docs))
        elif plan.action == "tool":
            if not allowed_tool(plan.tool, user_context):
                return refuse("The user is not authorized to call this tool")
            result = call_tool(plan.tool, plan.args, timeout=5)
            memory.append(("tool_result", result))
        elif plan.action == "answer":
            draft = llm_generate(build_answer_prompt(goal, memory))
            check = verify(draft, memory)
            if check.ok:
                return draft
            memory.append(("critique", check))
        else:
            return fallback("Unable to determine the next step")
    return fallback("Maximum number of steps reached; escalate to human review")

6. Evaluation, Observability, and Security Governance

RAG and agent systems should not be judged only by whether answers sound right. They need layered evaluation.

• Retrieval layer: recall, precision, ranking quality, and filter correctness.
• Generation layer: answer accuracy, faithfulness to the evidence, citation coverage, and refusal accuracy.
• Task layer: end-to-end success rate, average number of steps, tool call success rate, human intervention rate, and user feedback.
• Engineering layer: latency, cost, cache hit rate, and error rate.

Offline evaluation sets should cover common questions, long-tail questions, permission boundaries, and malicious inputs. Run them continuously as regression tests.

For observability, record the retrieval query, retrieved passages, reranking results, prompt version, model output, tool calls, and verification results for each request. This creates a traceable record of each request.

Security governance includes tenant isolation, permission checks, masking of sensitive information, content safety, audit logs, and cache invalidation policies. Retrieval filters must match the user’s permissions. The system must not retrieve all content first and remove sensitive information after generation.

7. Suggested Rollout Path

Start with high-value, low-risk, and measurable scenarios, such as internal knowledge Q&A, operations manual search, support ticket assistance, and report explanation.

First, build a minimal RAG pipeline and an evaluation set. Then add hybrid retrieval, reranking, citations, and refusal step by step. After the system runs reliably, let the agent handle query planning, tool calls, and multi-step tasks. Each stage should include budgets, fallback paths, and human confirmation.

The goal is not to build the most complex agent possible. The goal is a data connection system where evidence is traceable, permissions are enforced, results can be evaluated, and costs are manageable.