Choosing and Optimizing Large Language Models: Prompt Engineering, RAG, and Fine-Tuning

    A practical guide to selecting large language models based on business requirements, evaluating them with realistic task suites, improving prompts, choosing between RAG and fine-tuning, and managing cost, deployment, security, and production operations.

    1. Build a Capability Framework Before Asking Which Model Is Best

    Large language model selection is fundamentally a capability-matching problem under constraints. Business objectives, data security, response latency, budget, deployment model, and the engineering capabilities of the team all influence what “best” means. Decisions based only on personal impressions or isolated benchmark rankings often lead to a gap between impressive demos and unusable production systems. A more reliable approach is to establish a capability framework first, then validate candidate models against real-world task sets.

    Recommended evaluation dimensions include:

    1. Chinese language understanding and generation: Can the model accurately understand domain terminology, conversational language, ambiguous sentences, and long documents? Is its output stable, readable, and aligned with the tone of the industry?
    2. Reasoning and coding: How reliably does it handle multi-step reasoning, mathematical calculations, table understanding, code generation and explanation, and error diagnosis? Does it support structured output effectively?
    3. Multimodal capabilities: Does it meet content production requirements for text-to-image, image-to-text, video scripts, and image-text understanding? Is cross-modal consistency stable?
    4. Context and memory management: Does the model forget earlier assumptions during long conversations? How mature are its capabilities for long-document summarization, retrieval-augmented generation, and chunked processing?
    5. Tool calling and integration: Does it support reliable API calls, function-based tool orchestration, structured responses, permission isolation, and auditing?
    6. Deployment and compliance: Does it support private deployment, dedicated cloud environments, data masking, content safety, log auditing, and access control? Does it meet relevant industry requirements?
    7. Cost and reliability: Consider the cost per task, peak concurrency, p95 latency, failure-retry costs, and regression risks caused by model version upgrades.

    These dimensions do not have universal weights. Customer service applications typically prioritize latency, stability, and knowledge accuracy. Development scenarios place more emphasis on coding and reasoning. Content production values multimodal capabilities and writing style, while enterprise integration prioritizes security, compliance, and private deployment. Model selection should therefore document the weights in a reviewable evaluation table rather than relying on individual preferences.

    2. Three Common Capability Profiles and Their Boundaries

    Based on how capabilities are typically delivered, large model services can be broadly grouped into three profiles. These are not fixed categories, and actual boundaries may change with model versions and deployment methods. However, they provide a useful starting point for requirements discussions.

    2.1 Conversational and Content-Creation Models

    These models generally emphasize fast responses, natural dialogue, voice interaction, social media copy, and short-video script generation. They are suitable for daily assistance, language practice, lightweight office support, lifestyle information services, and initial content drafting. Their main strengths are quick onboarding, a low interaction barrier, and broad coverage of common use cases.

    Their limitations are also clear. Accuracy may be inconsistent for specialized academic questions, early assumptions can be forgotten during long conversations, domain depth may be limited, and complex multi-turn reasoning may be unreliable. When used for high-risk medical, financial, or legal questions, they must be combined with knowledge bases, retrieval-augmented generation, human review, and output validation. General conversational ability should not be treated as a professional conclusion.

    2.2 Enterprise Integration and Multimodal Generation Models

    These models typically emphasize enterprise services, cloud ecosystem integration, Chinese localization, multimodal generation, model fine-tuning, and private deployment. They are suitable for e-commerce content production, enterprise office applications, vertical-domain model development, business intelligence analysis, and multimodal content creation. Their advantages often include stronger security and compliance controls, service governance, and support for commercial deployment.

    However, the experience for individual users may not be optimal. Free quotas may be limited, paid usage requires careful cost analysis, response speed may depend on the service model, and the strength of the developer ecosystem should also be evaluated. Enterprise selection should consider more than model capability. Identity authentication, permission management, audit logs, data residency, service-level agreements, and incident response mechanisms are equally important.

    2.3 Reasoning and Coding Models

    These models generally focus on logical reasoning, mathematical computation, code generation and debugging, and optimization for specialized domains. Some open-weight models also support local deployment and lightweight adaptation. They are suitable for academic research, software development, reasoning tasks, professional question answering, and local deployment scenarios.

    Their weaknesses may include less engaging everyday conversations, relatively limited multimodal capabilities, weaker understanding of conversational Chinese than general-purpose interaction models, and a higher learning curve for new users. If a team lacks evaluation, fine-tuning, and operations expertise, direct private deployment may create significant hidden costs.

    3. Replace Subjective Impressions with Task-Suite Evaluation

    Model evaluation should reflect the distribution of production workloads as closely as possible. Teams can extract samples from historical support tickets, real user questions, repository issues, and content production tasks to build scenario-specific task suites with explicit pass criteria.

    At a minimum, evaluation should cover task success rate, structured-output compliance, factual consistency, p95 latency, cost per task, safety interception rate, and human review rate.

    Example evaluation configuration:

    1. {
    2. "suite": "customer_service",
    3. "cases": ["refund policy questions", "order status explanations", "multi-turn rescheduling", "sensitive requests routed to human agents"],
    4. "metrics": ["task_success_rate", "schema_valid_rate", "p95_latency_ms", "cost_per_task"],
    5. "pass_line": {
    6. "task_success_rate": 0.9,
    7. "schema_valid_rate": 0.98,
    8. "p95_latency_ms": 2500
    9. }
    10. }

    Three practices are especially important:

    1. Update offline evaluation sets regularly to detect regressions after model upgrades.
    2. Run controlled online rollouts and compare real user satisfaction, human handoff rates, and complaint rates.
    3. Add mandatory validation and human fallback processes for high-risk tasks.

    Model selection becomes explainable only when model capabilities are translated into measurable business outcomes.

    4. Prompt Optimization: Turn Requirements into an Executable Contract

    A prompt is not a magic spell. It is a contract that describes the task, context, constraints, and expected output format. A maintainable prompt template should define the role, task, input context, constraints, output structure, and a small number of examples. For structured responses, the application should require fixed fields and perform format validation and retries when necessary.

    Example prompt template:

    1. {
    2. "role": "enterprise knowledge assistant",
    3. "task": "Answer the user's question based on the provided materials",
    4. "context": "Use only the retrieved materials and do not fabricate information",
    5. "constraints": ["cite material IDs", "identify missing information when uncertain", "limit the response to 200 Chinese characters"],
    6. "output_schema": {
    7. "answer": "string",
    8. "citations": ["string"],
    9. "confidence": "high|medium|low"
    10. }
    11. }

    The phrase “200 Chinese characters” remains in the example because it reflects the original task requirement. In an English-language implementation, this constraint should be adapted to the target language and product requirements rather than changed arbitrarily.

    Recommended prompt optimization practices include:

    • Split long documents into searchable chunks, then summarize before answering.
    • For complex tasks, require checklists or verification points, but do not depend on unverifiable hidden reasoning traces.
    • Provide positive and negative examples for format-sensitive tasks.
    • Preserve summaries of key assumptions in multi-turn conversations.
    • Version prompts and run regression tests against the same evaluation suite.

    Prompt optimization can substantially improve results, but it cannot replace a knowledge base, access control, or business-level validation.

    5. Choosing Between Fine-Tuning, RAG, and Prompting

    Many teams consider fine-tuning immediately, but it is not always the best solution. A simplified decision framework is:

    • When knowledge changes frequently, private-domain knowledge is extensive, or source citations are required, prioritize retrieval-augmented generation.
    • When output style, formatting, terminology, or domain classification must be consistent, consider supervised fine-tuning or parameter-efficient fine-tuning.
    • When the task is complex but available examples are limited, optimize prompts and evaluate systematically first.
    • When cost is the primary concern, consider distillation, quantization, routing to smaller models, and caching.

    The key to fine-tuning is not the algorithm configuration but data quality. Training data should be cleaned, deduplicated, de-identified, and consistently labeled. Difficult examples and boundary cases should be retained.

    Example configuration:

    1. {
    2. "method": "supervised_finetune_low_rank",
    3. "base_model": "<approved_model>",
    4. "data_uri": "internal://dataset/v3",
    5. "rank": 8,
    6. "epochs": 3,
    7. "eval_suite": "domain_eval_v2",
    8. "guardrails": ["pii_redaction", "schema_validation"]
    9. }

    For enterprises without a dedicated engineering team, managed fine-tuning services or online fine-tuning platforms can reduce the cost of environment setup, training orchestration, and version management. However, teams must verify that data remains within the approved boundary, permissions are auditable, models can be rolled back, and evaluations are reproducible.

    The objective of customization is not to train a larger model. It is to solve a specific task more reliably and at a lower overall cost.

    6. Cost, Performance, and Deployment Models

    Cost optimization should begin with routing:

    • Assign simple classification, extraction, and rewriting tasks to smaller models or rule engines.
    • Route complex reasoning and coding tasks to more capable models.
    • Cache high-frequency questions.
    • Process batch workloads through asynchronous queues.
    • Compress long context before sending it to the model.

    Regarding deployment models, API access is suitable for rapid validation and elastic concurrency. Private deployment is better for sensitive data and deep customization. Hybrid deployment is appropriate when core data must remain local while non-sensitive workloads require cloud elasticity.

    Monitoring should cover request volume, failure rate, p95 latency, cost per task, output compliance, human review rate, and user satisfaction. Model upgrades should use staged releases and include rollback plans. Uncontrolled upgrades can introduce regressions into business workflows.

    7. Security, Compliance, and Governance

    Security governance should be addressed before large models reach production. A practical approach is to classify data into public, internal, sensitive, and critical categories, then define the permitted models, masking policies, and audit requirements for each category.

    Additional controls should include:

    • Detect and mask sensitive information in inputs and outputs.
    • Apply least-privilege access to tool calls.
    • Add rule-based validation and human review for high-risk outputs.
    • Retain invocation logs, prompt versions, model versions, and evaluation records.

    When managed model services, vector retrieval, object storage, logging and monitoring, or container platforms are required, organizations may use general cloud capabilities from compliant providers such as Baidu AI Cloud to build the infrastructure. The specific provider and service should be evaluated carefully for data residency, permission auditing, service-level agreements, and incident response. These checks are necessary to ensure alignment with enterprise governance requirements.

    8. An Implementation Roadmap

    A four-stage rollout is recommended:

    1. Scenario selection and evaluation set construction: Use two to four weeks to identify priority scenarios, build evaluation suites, and define pass criteria.
    2. Baseline and controlled experiments: Establish a prompt baseline, add retrieval augmentation, and run small-scale comparative experiments to validate business value.
    3. Staged production rollout: Release the system gradually and connect monitoring, auditing, and human fallback mechanisms.
    4. Continuous optimization: Use production data to refine prompts, knowledge bases, and fine-tuning strategies, establishing a versioned operating process.

    Each stage should produce concrete deliverables, including an evaluation report, cost analysis, risk register, and rollback plan.

    9. Conclusion

    Large language model selection should not stop at the subjective question, “Which provider is easiest to use?” It should become a more precise question: “Under the given constraints, which combination of capabilities can reliably complete the target tasks?”

    Scenario matching determines the direction. Prompt optimization establishes the baseline. Retrieval augmentation and fine-tuning provide professional depth. Cost management and security governance determine whether the system can scale.

    Replacing one-time decisions with task-suite evaluation, staged releases, and continuous monitoring is the foundation for bringing large language models into reliable production systems.