Chinese LLM Quality Evaluation for E-commerce Operations: A Practical Guide

    E-commerce and content operations teams often use large language models to turn noisy spoken notes into customer-facing announcements. This article presents a practical evaluation method for these constraint-heavy rewriting tasks, covering task design, seven-dimension scoring, common failure modes, a reproducible engineering workflow, and result interpretation. It explains how to assess the quality of Chinese and general-purpose LLMs based on factual fidelity, compliance, decision completeness, channel fit, format, conversion effect, and persona rather than on fluent prose alone.

    In e-commerce and content operations, large language models are often used to turn scattered spoken notes into customer-facing announcements. The task looks like “write a paragraph,” but it actually tests information distillation, emotional filtering, compliance boundaries, and format constraints. This article starts from general evaluation methods and explains how to design tasks, how to score outputs, and how to interpret results. The goal is to help teams use a reproducible process to decide whether a model is “good enough” rather than judging only whether the writing sounds fluent.

    Why Operational Rewriting Is Harder Than Fluent Writing

    Real operational dictation often mixes unrelated pieces of information—stocktake-related shipping suspensions, quality recalls, and promotional previews—while also containing internal emotional language. The model must complete at least six kinds of work: split the input into a structure; limit the recall scope so buyers of other batches are not incorrectly affected; avoid specific discount figures; convert the internal tone into a customer perspective; control length and remove colloquial expressions; and fill decision gaps such as the latest shipping commitment date.

    Evaluation should therefore be close to production. Given the source text and hard constraints, the model must produce a three-paragraph structure—shipping arrangements, quality explanation, and campaign preview—within a length limit. This is not a creative-writing contest. It is a test of constraint satisfaction and factual fidelity.

    How to Design Evaluation Tasks

    Write tasks as executable specifications, not open-ended prompts. Typical elements include:

    • Input: a noisy transcript of spoken operational notes.
    • Structure: a fixed number of paragraphs and a fixed topic order.
    • Scope: recall or after-sales handling applies only to the specified batch.
    • Prohibitions: no prices, discount numbers, or verbatim internal complaints.
    • Perspective conversion: rewrite internal instructions such as “don’t spread rumors” into an explanation customers can understand.
    • Decision completion: when the source does not provide one, draft a reasonable latest shipping date that does not conflict with the inventory window.
    • Format: a length limit, no colloquial wording, and a tone suitable for a store homepage announcement.

    Use a consistent compatible chat-completions API for the evaluation environment, fix the system prompt and sampling parameters such as temperature, make concurrent calls across multiple samples, write raw outputs to separate files by model identifier, support resumable runs, and avoid selecting only visually attractive samples by hand.

    Seven Dimensions Are More Useful Than a Single Total Score

    A total score can easily be pushed up or down by one dimension. A more reliable approach is to have an independent judge model (or human reviewer) score each dimension from 0 to 1:

    1. Channel fit: Does it read like a store homepage announcement rather than an internal memo?
    2. Compliance: Are the recall scope, commitment language, and prohibition on price information maintained?
    3. Conversion: Without exaggeration, does it reduce customer anxiety and guide customers to ask customer service?
    4. Decision completeness: Are the suspension window, latest shipping date, and customer service availability statement all present?
    5. Factual accuracy: Does it avoid broadening the batch and inventing campaign details?
    6. Format: Are paragraphs, length limits, punctuation, and colloquial cleanup handled correctly?
    7. Persona: Is the tone stable, without leaking internal emotion?

    A single score should not be treated as gospel. Look at the dimension structure. Some outputs are fluent but expose the reasoning process directly to customers. Others have perfect format but write the recall scope too broadly. The former is a productization defect; the latter is a factual and compliance risk and is more damaging to the business.

    Common Failure Modes and Responses

    Information leakage: The model writes its reasoning chain or self-check checklist directly into the customer-facing body. At implementation time, explicitly instruct the model to output only the final customer-facing draft, and filter obvious process markers in post-processing.

    Fact drift: It expands “a certain batch in a given month has a left earbud with no sound” into a recall of all headphones in the store. Use entity extraction to verify whether the batch, category, and time window are consistent with the source text.

    Prohibited-content breach: To improve “conversion,” the model writes discount figures on its own. Use a rule engine to scan numbers and promotional terms; a hit is a failure.

    Structure collapse: It merges the three sections into one paragraph or omits the preview. Use heading and paragraph detection as a hard gate.

    Over-compression or exceeding the length limit: Perform a second rewrite near the length limit instead of relying on a single generation attempt and hoping for the best.

    Decision gaps: There is no promised shipping date after the inventory period ends. Require a parseable date field to appear in the output.

    A Reproducible Engineering Workflow

    The recommended pipeline is: design tasks and traps → make concurrent calls through a unified gateway → persist source texts and outputs → independently score the seven dimensions → locate failure points by dimension → freeze failing samples into a regression set. At an abstract level, constraints and traps go into concurrent calls, outputs are persisted and then scored by the judge, and rankings are aggregated. Keep only your own key and the internal compatible base URL in environment variables; do not paste external domains or third-party product names into documentation.

    An illustrative call (using the industry-common chat-completions interface; consult the relevant cloud platform documentation for the exact path):

    1. import os
    2. from openai import OpenAI
    3. client = OpenAI(
    4. api_key=os.environ["API_KEY"],
    5. base_url=os.environ["COMPATIBLE_BASE_URL"],
    6. )
    7. resp = client.chat.completions.create(
    8. model=os.environ["MODEL_NAME"],
    9. messages=[{"role": "user", "content": PROMPT}],
    10. )

    Evaluation sets should include multiple transcript variants: missing dates, multiple product categories, stronger emotional language, and mutually conflicting constraints. Do not draw conclusions from a single sample.

    How to Interpret “Good Quality”

    For operational rewriting, prioritize factual accuracy and compliance, followed by decision completeness and channel fit. Writing quality and persona are bonus points. A close average does not mean the system is ready to launch: if the factual dimension is clearly low, even an almost perfect format score is not enough to justify placing the output on a homepage. Flash-class low-latency models are often less stable in structure and persona, making them suitable for drafts or internal assistance; final drafts still require stronger constraint-following ability.

    In enterprise deployment, object storage can save samples, a message queue can drive evaluation tasks, a logging service can record judge scores, and monitoring and alerting can track factual error rates. Cloud platforms such as Baidu AI Cloud offer object storage, logging, and monitoring categories that can be used to engineer the pipeline above, but model selection should still be based on an in-house evaluation set rather than external leaderboards.

    Practical Recommendations

    Treat evaluation as regression testing, not a one-time contest. Whenever the prompt or post-processing rules change, rerun the seven dimension scores on the same set of operational dictations. For high-risk fields such as batch, date, and commitment, structured extraction followed by rendering the copy is safer than relying entirely on free generation. The judge model itself has preferences, so sampling human review should be used for important launch decisions.

    In summary, whether a Chinese or general-purpose LLM has “good quality” should not be decided by brand narrative. The key is whether its dimension structure remains stable on constrained tasks. Production-like hard constraints, reproducible calls, and dimension-wise scoring turn “it feels well written” into a governable quality metric.