Large Language Models in Production: An Engineering Guide to Chat, Code Generation, Multimodal Reasoning, and RAG

Introduction: From “Demoable” to “Deliverable”

Many teams evaluate large language models based on demonstrations such as chatting, writing code, or analyzing images. However, the real challenges appear after deployment: how to make multi-turn conversations stable, how to avoid common pitfalls in code generation, how to evaluate multimodal reasoning, how to control hallucinations, how to reduce costs, and how to isolate permissions. The value of large language models does not come from a single point feature, but from whether general capabilities can be turned into a reliable system aligned with business needs.

This article focuses on intelligent conversation, code generation, multimodal reasoning, and engineering implementation. It discusses capability boundaries and also provides executable templates, review checklists, and quantitative metrics. The emphasis is not on whether a model “can be used,” but on how to choose, integrate, evaluate, and maintain it.

1. Intelligent Conversation: Control Context First, Then Optimize Multi-Turn Experience

Intelligent conversation is one of the most visible capabilities of large language models. However, when moving to production, the main risks are usually not whether the model can chat, but whether context is controllable, whether key information is preserved, and whether costs are sustainable.

1.1 From Keyword Matching to Semantic Understanding

Traditional search or rule-based bots often rely on keyword matching. If users change their wording, the system may fail to understand them. Large language models use pretraining and contextual understanding to make reasonable inferences about vague expressions, colloquial language, and typos. For example, if a user says “translate this English paragraph into Chinese and highlight the key points,” the model can identify both tasks—translation and summarization—and return a combined result.

Semantic understanding also has limits. For highly specialized tasks, tasks that depend heavily on real-time data, or tasks that require exact factual accuracy, the model’s internal knowledge alone is not enough. A common approach is to supplement facts with retrieval-augmented generation, tool calling, or rule-based validation.

1.2 Multi-Turn Conversation: Context Management and Fallback Strategies

The key to multi-turn conversation is not simply inserting all history into the context window, but structurally preserving goals, constraints, facts, and user preferences.

It is recommended to divide conversation history into four categories:

1. System settings: role, tone, and safety boundaries.
2. User goals: the current task and final deliverable.
3. Key facts: confirmed data, constraints, and reference materials.
4. Recent interactions: the latest few exchanges, used to maintain continuity.

A common engineering approach is to keep recent turns and compress earlier turns with summaries:

def compress_history(history, keep_recent=6, token_limit=4000):
    # token_count can be implemented using the actual tokenizer
    if token_count(history) <= token_limit:
        return history, False
    recent = history[-keep_recent:]
    early = history[:-keep_recent]
    try:
        summary = summarize(early)  # call the summarization model
        compressed = [
            {"role": "system", "content": "Historical summary: " + summary},
            {"role": "system", "content": "Key information: user role, constraints, and confirmed facts; mark as unconfirmed if missing."}
        ]
        return compressed + recent, True
    except Exception:
        # Fallback if summarization fails: keep the first and last key snippets to avoid completely losing context
        fallback = early[:1] + early[-1:]
        return fallback + recent, True

Four engineering details need extra attention:

• Summarization cost: summarization may consume extra tokens, add latency, and increase cost. Set a trigger threshold instead of summarizing every turn.
• Fallback for missing key information: if summarization fails, at least preserve system instructions, core user goals, and recent key facts.
• Context token limit: a longer context window is not always better. Cost, latency, and attention decay should also be considered.
• Basis for retaining recent turns: keeping the most recent 4 to 8 turns is common, depending on task complexity, user confirmation frequency, and information density per turn.

1.3 Evaluating Conversation Quality

Do not judge a conversational system only by whether the response sounds fluent. Build a small evaluation set covering the following samples:

• Vague expressions: omitted subjects, typos, or colloquial descriptions.
• Multi-turn follow-ups: users add constraints based on prior context.
• Boundary-crossing questions: issues involving privacy, permissions, or sensitive information.
• Factual questions: cases requiring reference materials or tool results.

Quantifiable metrics include intent recognition accuracy, key-information retention rate, user follow-up rate, number of erroneous clarification attempts, P95 latency, and per-turn token cost.

2. Automated Code Generation: From Prompting to a Review Loop

Code generation is one of the most common ways to apply large language models in engineering. However, it is better treated as a “high-quality starting draft” than as a final deliverable. A practical code generation workflow should include prompting, generation, review, testing, and gradual release.

2.1 Three Levels of Code Generation

1. Snippet-level completion: completing a small amount of code based on comments, variable names, or function signatures.
2. Function-level generation: generating a complete function, script, or component from a requirement description.
3. Engineering-level assistance: generating tests, API documentation, configuration templates, refactoring suggestions, or troubleshooting ideas.

Different levels require different review intensity. Snippet completion can be checked quickly by a person. Function-level generation requires unit tests. Engineering-level assistance requires code review, security scanning, and gradual release.

2.2 High-Quality Prompt Template: Context, Constraints, and Expected Output

The quality of code generation depends heavily on how clearly the input is described. It is recommended to use a structure of “context + constraints + expected output,” and to specify dependency versions, exception handling, logging fields, and the supported execution scope.

The following is a directly runnable example for generating an HTTP request wrapper with retry logic. It depends only on the Python standard library and works well as a minimal runnable template.

import logging
import time
import urllib.error
import urllib.request
from typing import Optional
RETRYABLE_STATUS = {408, 429, 500, 502, 503, 504}
def request_with_retry(
    url: str,
    max_retries: int = 3,
    timeout: int = 5,
    backoff_factor: float = 0.5
) -> bytes:
    """
    Send an HTTP GET request using the Python standard library urllib.
    - max_retries: maximum number of retries, default 3.
    - timeout: timeout for each request, in seconds.
    - backoff_factor: exponential backoff base.
    - Retry only on network errors and retryable status codes.
    """
    last_error: Optional[Exception] = None
    for attempt in range(max_retries + 1):
        started = time.monotonic()
        try:
            req = urllib.request.Request(
                url,
                headers={"User-Agent": "retry-http-client/1.0"}
            )
            with urllib.request.urlopen(req, timeout=timeout) as resp:
                elapsed_ms = int((time.monotonic() - started) * 1000)
                logging.info(
                    "http_request",
                    extra={
                        "attempt": attempt,
                        "url": url,
                        "status": resp.status,
                        "elapsed_ms": elapsed_ms,
                    },
                )
                return resp.read()
        except urllib.error.HTTPError as e:
            last_error = e
            elapsed_ms = int((time.monotonic() - started) * 1000)
            logging.warning(
                "http_error",
                extra={
                    "attempt": attempt,
                    "url": url,
                    "status": e.code,
                    "error_type": "HTTPError",
                    "elapsed_ms": elapsed_ms,
                },
            )
            if e.code not in RETRYABLE_STATUS:
                raise
        except (urllib.error.URLError, TimeoutError) as e:
            last_error = e
            elapsed_ms = int((time.monotonic() - started) * 1000)
            logging.warning(
                "network_error",
                extra={
                    "attempt": attempt,
                    "url": url,
                    "status": None,
                    "error_type": type(e).__name__,
                    "elapsed_ms": elapsed_ms,
                },
            )
        if attempt >= max_retries:
            break
        time.sleep(backoff_factor * (2 ** attempt))
    raise RuntimeError(f"request failed after {max_retries} retries") from last_error

This example covers common engineering parameters:

• Timeout: timeout controls the duration of each request.
• Retry: retry up to 3 times by default.
• Backoff: exponential backoff reduces transient pressure.
• Exception classification: distinguish HTTPError, URLError, and TimeoutError.
• Logging fields: attempt, url, status, error_type, and elapsed_ms.

If connecting to a real business system, additional items should be added: POST request bodies, authentication headers, proxy settings, connection pools, circuit breakers, tracing IDs, and idempotency keys.

2.3 Post-Generation Review Checklist

After generating code, check it against the following checklist instead of merging it directly:

1. Correctness check: boundary conditions, exception branches, timeout, and retry logic are complete.
2. Unit tests: normal paths, exception paths, and retry paths are covered.
3. Static analysis: syntax errors, unused variables, dangerous functions, and excessive complexity are detected.
4. Dependency version locking: unnecessary third-party libraries are avoided, and dependencies are recorded in a lock file.
5. Secret scanning: tokens, passwords, certificates, or internal addresses are not hard-coded.
6. Security scanning: injection, SSRF, unsafe deserialization, and similar risks are checked.
7. Gradual release: small traffic validates the change before full rollout.
8. Human review: high-risk logic involving payments, permissions, deletion, or data export must be confirmed by a person.

A mature criterion is this: generated code can reduce the time cost of writing from scratch, but it cannot replace testing, review, and security processes.

3. Multimodal Reasoning: Guidance, Evaluation, and Validation

Multimodal reasoning does not simply mean looking at text and images at the same time. It refers to a model’s ability to jointly understand cross-modal information such as text, images, tables, charts, and scanned documents, and then produce conclusions based on the task. Different models support multimodal capabilities to different degrees, so model capabilities should be confirmed before integration.

3.1 Rigorous Reasoning vs. Divergent Generation: Two Task Modes

Large language models can be understood as systems that can be guided into different task modes through prompts:

• Rigorous reasoning mode: suitable for mathematical calculations, code debugging, causal analysis, table verification, and process review, with emphasis on verifiability.
• Divergent generation mode: suitable for copywriting, solution design, brainstorming, and creative expansion, with emphasis on diversity and readability.

Many tasks require rigorous reasoning first and divergent generation later. For example, when writing a technical solution, one can first organize architecture dependencies and risks, then compose the expression. The prompt can specify the mode:

First, use rigorous reasoning mode to list all known conditions and constraints, then derive the conclusion step by step.
If a creative proposal is needed, label it separately as "Creative suggestions" and explain how it differs from the rigorous conclusion.

3.2 How to Use Chain-of-Thought: Applicable Conditions and Boundaries

For complex reasoning tasks, the model can be asked to show intermediate steps, such as listing known conditions, decomposing sub-problems, calculating step by step, and finally validating units and boundaries. This approach may reduce errors in complex calculations and multi-step logic, and it also makes human review easier.

However, showing the reasoning process does not guarantee factual correctness. Chain-of-thought is more suitable when:

• The task can be split into multiple steps.
• Intermediate results can be checked by a person.
• The model’s capability matches the task complexity.
• External fact sources, tool results, or computational rules constrain the process.

If a task cannot be decomposed, has no validation criteria, or clearly exceeds the model’s capability, reasoning steps alone cannot guarantee correctness. In such cases, introduce rule engines, calculators, retrieved materials, or human review.

3.3 Evaluating Multimodal Reasoning

Multimodal tasks should not be evaluated only by subjective judgment. Build a layered evaluation set:

1. Text understanding: summarization, classification, and question answering.
2. Image understanding: OCR, chart recognition, and scene description.
3. Table understanding: field extraction, numerical calculation, and cross-table association.
4. Cross-modal joint reasoning: judging conclusions based on both images and text.
5. Security testing: detecting attempts to induce privilege escalation, sensitive information leakage, or misleading materials.

Quantifiable metrics include:

• Field extraction accuracy.
• Table calculation accuracy.
• OCR character accuracy.
• Agreement rate between image descriptions and human ratings.
• Unsupported answer rate.
• Correct refusal rate.
• Human sampling pass rate.
• P95 latency and per-request cost.

The evaluation process should use “automated metrics + double-blind human sampling + error attribution.” Automated metrics are used for fast filtering, while human sampling is used to discover semantic errors, hallucinations, and boundary risks.

4. Engineering Implementation: Using Metrics, Templates, and Processes to Reduce Uncertainty

After large language models enter production, the real difficulty is usually not a single call, but sustained stability. Engineering implementation should build closed loops around retrieval augmentation, prompt management, evaluation and monitoring, cost and performance, and security and compliance.

4.1 Retrieval-Augmented Generation: Making Answers Evidence-Based

Retrieval-augmented generation (RAG) is commonly used for enterprise knowledge-base question answering, customer support assistance, document retrieval, and internal assistants. The basic flow is:

Document collection -> cleaning -> chunking -> embedding -> storage -> retrieval recall -> reranking -> context assembly -> answer generation -> citation display

Recommended metrics include:

• Recall rate: whether the relevant snippets are retrieved.
• Citation accuracy: whether the cited snippets support the final answer.
• Unsupported answer rate: whether the answer goes beyond the retrieved materials.
• Refusal rate when materials are insufficient: whether the system clearly says it cannot answer.
• Permission hit rate: whether users retrieve only data they are authorized to access.

The answer template can require the model to strictly rely on retrieved snippets:

Answer only based on the following materials.
If the materials are insufficient, output "Insufficient materials".
At the end of the answer, mark the citation numbers. Do not fabricate conclusions that are not present in the materials.

RAG also needs to prevent “fake citations.” Citation numbers should correspond one-to-one with retrieved snippets, and the original source should be displayed in the frontend.

4.2 Prompt Engineering: Manage Prompts Like Code

Prompts should not exist only in a chat window; they should be placed under version control. It is recommended to manage the following:

• Prompt version number.
• Supported model.
• Input variables.
• Output format.
• Evaluation samples.
• Failure cases.
• Regression results.

A simple A/B test can be set up:

1. Fix a business sample set.
2. Run the old prompt and the new prompt on the same questions.
3. Compare accuracy, refusal rate, latency, and cost.
4. Promote to gradual release only after meeting preset thresholds.

4.3 Evaluation and Monitoring: Offline Evaluation + Online Alerts

Before launch, an offline evaluation set is needed. After launch, online monitoring is required. Recommended metrics include:

• Answer accuracy or human rating.
• Refusal rate and miss rate.
• P95 latency.
• Per-request cost.
• Token consumption.
• Retry rate.
• Retrieval hit rate.
• Sensitive-word hit rate.
• Unauthorized access blocking rate.
• Hallucination complaint rate.

A recommended troubleshooting flow is:

1. Determine whether the issue belongs to input, retrieval, model, or output presentation.
2. Check whether the input contains unauthorized data, sensitive data, or malicious prompts.
3. Check whether retrieval returns the correct materials and whether permission filtering is correct.
4. Check whether the model output exceeds the materials or violates the output format.
5. Check whether latency and cost increase due to overly long context, excessive retries, or retrieval anomalies.
6. Trigger rollback, rate limiting, or human intervention for high-risk errors.

4.4 Cost and Performance Balance: Do Not Use the Most Powerful Model for Every Task

Cost optimization can start from four directions:

• Cache high-frequency results: cache repeated questions, fixed templates, and stable document question answering.
• Tiered routing: use lightweight models for simple classification, extraction, and formatting tasks; use stronger models for complex reasoning and code review.
• Context compression: reduce irrelevant history, redundant materials, and unnecessary system prompts.
• Batch processing: merge offline task requests to reduce the number of calls.

Recommended metrics include cache hit rate, average latency, P95 latency, per-request cost, token cost, retry cost, and human review cost.

4.5 Security and Compliance: Permissions, Masking, and Auditing

After large language models connect to enterprise data, security and compliance should be treated as default capabilities rather than post-hoc patches. At a minimum, include the following:

• Input side: sensitive information detection, prompt injection detection, and blocking unauthorized questions.
• Output side: sensitive information filtering, citation source validation, and blocking illegal or harmful information.
• Data side: sensitive data masking, least-privilege access, and export approval.
• Audit side: record request time, user, model version, retrieval source, response summary, and error code.
• Permission side: isolate knowledge bases by role to avoid mixing data across departments.
• Anti-deletion and leak prevention: set up a recycle bin, access allowlists, double confirmation for bulk deletion, and alerts for abnormal export.

Use metrics such as sensitive-word hit rate, unauthorized access blocking rate, masking coverage, and audit completeness for continuous monitoring.

Conclusion: Turning General Capabilities into a Stable Solution

The capabilities of intelligent conversation, code generation, and multimodal reasoning in large language models are not isolated. Together they form the capability stack of intelligent applications. However, whether they create real value depends on engineering design. Conversation systems need context governance. Code generation needs a review loop. Multimodal reasoning needs evaluation and factual constraints. The overall system needs metrics, monitoring, permissions, and cost control.

For developers and decision-makers, the more important question is not whether a particular technology is used, but whether clear capability boundaries, reusable evaluation mechanisms, and traceable governance processes have been established. Integrating large language models into business processes can improve efficiency and intelligence, but the effect usually depends on model capability, data quality, scenario complexity, and evaluation mechanisms. Only by turning general capabilities into a stable solution aligned with business needs can uncertainty be reduced and continuous benefits be achieved.