Evaluating the Business Writing Capabilities of Six LLMs Through a 618 Campaign Retrospective

    This article uses a structured e-commerce campaign retrospective to evaluate six large language models on business writing. It explains the test design, reproducibility requirements, seven-dimensional scoring framework, observed failure patterns, limitations, and practical guidance for enterprise model selection and evaluation.

    1. Why Use a Retrospective Report as the Test Task

    Everyday conversations with large language models have become highly fluent. Most models can write poems, chat, and explain concepts. The real differences emerge when a task contains multiple constraints and the model must satisfy both explicit and implicit requirements.

    Business writing is a typical example. A report must address the information needs of business decision-makers, present both achievements and problems, and follow a predefined structure.

    This evaluation used a project retrospective for a major e-commerce promotion as the test task. The task contained four major challenges:

    1. Balancing multiple objectives. The project went live three days late, but its conversion rate exceeded the target by 20%. The report had to present both facts. It could not use business performance to conceal the delay, nor could it allow the delay to overshadow the results.

    2. Keeping attribution within reasonable boundaries. On the first day, an incorrect discount price was displayed for two hours, affecting approximately 300 orders. The report needed to attribute the problem to process and operational mechanisms rather than to a specific individual.

    3. Accounting for resource constraints. During the project, two team members were reassigned to support another project for three days. This fact needed to receive appropriate weight in the delay analysis and could not simply be attributed to a design revision.

    4. Following a rigid structure. The report had to be organized into four sections: Results Review, Keep, Problem, and Try. Actions in the Try section needed to be specific and executable. Generic phrases such as “improve communication” or “reflect deeply” did not meet the requirements.

    Together, these constraints simulate the complexity of real-world business requests. The model had to understand instructions, select relevant information, and express it in a structured way. A weakness in any one of these capabilities could affect the final report.

    2. Test Methodology

    2.1 Overall Process

    The evaluation followed this workflow:

    1. Define the task.
    2. Call six models concurrently.
    3. Save the outputs locally.
    4. Use an independent judge model to score all outputs.
    5. Aggregate the scores and produce a ranking.

    The process was designed to make the results reproducible, auditable, and traceable. All models received the same prompt, each model generated one report, and the judge model scored each report once.

    2.2 Calling Method and Reproduction Conditions

    The models were called through OpenAI-compatible interfaces. The core code was as follows:

    1. import os
    2. import time
    3. from openai import OpenAI
    4. client = OpenAI(
    5. api_key=os.environ["MODEL_API_KEY"],
    6. base_url=os.environ["MODEL_BASE_URL"],
    7. timeout=120,
    8. )
    9. def call_model(model_id: str, prompt: str, retries: int = 3) -> str:
    10. for attempt in range(1, retries + 1):
    11. try:
    12. resp = client.chat.completions.create(
    13. model=model_id,
    14. messages=[{"role": "user", "content": prompt}],
    15. # Record the sampling parameters here for reproducibility.
    16. )
    17. choice = resp.choices[0]
    18. # Treat any finish_reason other than stop as truncated output.
    19. if choice.finish_reason == "stop" and choice.message.content:
    20. return choice.message.content
    21. raise RuntimeError("Output was empty or truncated")
    22. except Exception:
    23. if attempt == retries:
    24. raise
    25. time.sleep(2 ** attempt)

    Using the same prompt and interface reduces differences at the input layer, but it cannot eliminate model randomness or network and service-side variation. To reproduce the evaluation, the following parameters should be recorded:

    • Model name and version
    • temperature
    • top_p
    • seed, if supported by the interface
    • Maximum output length
    • Concurrency level
    • Timeout
    • Number of retries

    The specific values used in this test are not disclosed. Teams conducting their own evaluations should record and fix these parameters.

    2.3 Seven-Dimensional Scoring

    The judge model scored each report across seven dimensions, with each dimension receiving a score from 0 to 1:

    • Audience fit: Whether the report meets the information needs of business decision-makers and is easy to understand.
    • Compliance: Whether the report follows the explicit rules and implicit boundaries in the prompt.
    • Factual accuracy: Whether the data, timeline, and causal relationships match the source material.
    • Format compliance: Whether the report follows the required four-part structure and uses clear heading levels.
    • Language quality: Whether the wording is accurate and concise, without empty or formulaic expressions.
    • Persuasiveness: Whether the reasoning is rigorous and the attribution of causes is credible.
    • Information selection: Whether key information is prioritized appropriately and major omissions are avoided.

    For each dimension, it is useful to define what scores of 0, 0.5, and 1 mean. Clear scoring anchors can reduce subjective variation in judging. Compared with a single overall score, seven separate dimensions provide a clearer view of each model’s strengths and weaknesses.

    3. Results and Limitations

    3.1 Limitations First

    These results come from a small exploratory test based on a single sample from each model. Because every model generated only one report, the results reflect differences in this specific test and should not be generalized into universal conclusions.

    The judge model may also have its own scoring preferences. This evaluation did not include cross-validation with multiple judge models.

    3.2 Score Overview

    The average score of the top-performing group was 0.92, while the lowest-performing group averaged 0.61. The difference between the two averages was 0.31 points.

    This 0.31-point difference is not the same as the range across individual dimensions. For example, in factual accuracy, the highest score was 0.98 and the lowest was 0.28, producing a range of 0.70.

    Model Tier Average Score Range Representative Characteristics
    Top tier 0.85–0.92 Relatively balanced across dimensions, with stable factual accuracy
    Middle tier 0.68–0.77 Strong in some dimensions but with clear weaknesses
    Lower tier 0.61 and below Serious deficiencies across multiple dimensions

    The average score in the table is the mean of the seven dimension scores for each model.

    3.3 Three Common Failure Patterns

    The following examples are illustrative. They explain the types of failures that may occur and are not excerpts from the measured outputs.

    Pattern 1: Truncated output. The report stops halfway through and has no conclusion. This may be a minor issue in a demonstration, but in a business delivery it makes the content unusable. The finish_reason check in the code above can detect this type of failure.

    Pattern 2: Forgotten implicit rules. The first half of the report follows the requirements, but the second half attributes the entire delay to the design revision and ignores the reassignment of team members. This type of failure is more common in long-form, multi-constraint tasks.

    Pattern 3: Factual deviation. The report has a complete structure and fluent language, but contains incorrect key figures. For example, it may change a conversion rate of 4.2% to 4.0%, or rewrite “approximately 300 affected orders” as “more than 300 orders.” Such errors can mislead business decisions.

    4. How Enterprises Can Use This Method

    4.1 Configuring Weights for Factuality and Instruction Following

    Model selection should start with the organization’s specific use cases. For long-document generation and data analysis reports, factual accuracy and instruction following should receive higher weights. For marketing copy, language quality and persuasiveness may deserve greater emphasis.

    There is no single weighting scheme that works for every business scenario.

    4.2 Tiered Deployment and Human Review

    A tiered deployment strategy with human review is recommended. High-stakes tasks can be assigned to top-tier models, while highly structured texts with limited creative requirements can be assigned to mid-tier models to balance cost and output quality.

    Regardless of the model tier, key figures in a report should be verified by a human or an independent tool before entering a formal workflow. Factual weaknesses are a risk that can affect all types of models.

    From a cost perspective, a report that requires two hours of manual editing may still be useful, while a completely unusable draft has little value. The difference in business value is often much greater than the difference in model invocation cost. As a result, output quality generally deserves more attention than the price of a single call.

    5. How to Design Your Own Evaluation

    Test tasks should be sufficiently challenging. They should include multiple objectives, implicit constraints, formatting requirements, and a threshold for factual accuracy. Tasks that are too simple may produce similar scores for every model and provide little support for model selection.

    Teams should consider the following practices:

    • Sample each model multiple times and calculate the average.
    • Use multiple judge models or add manual spot checks.
    • Define clear criteria for scores of 0, 0.5, and 1 for every dimension.
    • Record model versions and sampling parameters so the evaluation can be reviewed and rerun.