Evaluating Next-Generation LLMs: A Practical Review on Writing Tasks
This article reviews the practical performance of next-generation Large Language Models (LLMs) in writing tasks. It explores the significant shift in token consumption, the challenges of over-defensive constraint adherence, and the impressive "human-like" text generation capabilities evaluated by AI detectors. Furthermore, it offers engineering strategies for cost optimization, interaction friction reduction, and a decision-making checklist for developers considering an upgrade.
1. Test Background and Methodology
After the release of next-generation large language models, the most common questions from developers are whether the upgrade is worth it and whether it will cost significantly more. This article first outlines the test setup to clarify the scope of the following conclusions.
- Baseline Model: The previous-generation flagship model (referred to as the “previous model”), using its daily workflow performance as a reference.
- Candidate Model: The next-generation flagship model (referred to as the “new model”), with version differences unified under this term.
- Task Scale: Five independent but structurally similar tasks, all driven by long prompts exceeding a thousand words for content generation.
- Generation Tier: The highest reasoning tier provided by the model was enabled for time-sensitive tasks.
- Output Quality Assessment: Borrowing the focus on “human degree” from the LLM writing community, mainstream AI content detection tools were used to score and evaluate the results.
2. Token Consumption: Where Does the Order-of-Magnitude Jump Come From?
2.1 Phenomenon: The Same Five Tasks Consume Nearly Ten Times More Tokens
Switching daily tasks from the previous model to the new model resulted in an order-of-magnitude jump in token consumption. On the previous model, these five tasks consumed about 1%–2% combined. On the new model, with the highest reasoning tier enabled, the five tasks consumed about 18%. The difference is roughly 9–10x, partly due to input length: long prompts are repeatedly counted as input in each interaction round.
Two concepts need clarification here. The “reasoning tier” can be understood as the model’s “depth of thought” or reasoning budget tier, determining how much computation the model is willing to invest in breaking down problems and verifying hypotheses before giving a final answer. The “Ultra tier” can be understood as “maximizing the reasoning budget,” trading more compute for complex reasoning capabilities at the cost of longer response times. The growth in token consumption is essentially the model performing more “internal verifications” for the same question.
2.2 Causal Analysis: Why Writing Tasks Are Especially Token-Heavy
- The “Thinking Cost” of Long Chain-of-Thought: The token consumption of the previous model basically corresponded to “input + output.” The new model expands a longer reasoning process before generating an answer, which is also counted as consumption. Developers of reasoning models typically label “reasoning tokens” separately in official billing instructions. The “reasoning budget” in the illustrative formula in Section 5 corresponds to this type of consumption.
- The “Amplification Effect” of Multi-Turn Confirmations: Each user confirmation is equivalent to initiating a complete round trip: system prompts, constraint lists, and task descriptions are all resent, the model reasons again, and the user’s confirmation is appended to the context. The more confirmation rounds, the more severe the repeated computation.
- The Overlap of Long Context and Multi-Task Parallelism: Five tasks running simultaneously cause context to accumulate continuously. Each task undergoes several rounds of confirmation, further driving up the cost per task.
- Differences in Default Model Behavior: The new model is more cautious in instruction following (see Section 3). Under the same prompt, it may generate longer drafts, self-check explanations, and even changelogs.
2.3 Methodology: Establish a “Unit Task Cost” for Each Task
Asking whether a single query is “expensive” is meaningless; comparable cost baselines should be established by task type:
def task_cost(task: dict, price: dict) -> dict:# total_in: total input + cache miss + reasoning tokens# cached_in: tokens hitting prefix cache, billed at a lower pricetotal_in = task["total_in"]cost_in = (total_in - task["cached_in"]) * price["in"] \+ task["cached_in"] * price["cached_in"]cost_out = task["total_out"] * price["out"]return {"cost": cost_in + cost_out,"cost_per_1k_output": (cost_in + cost_out) / (task["total_out"] / 1000),"reasoning_ratio": round(task["reasoning_out"] / task["total_out"], 3),}
The illustrative code above highlights the core metric: cost per unit of valid output, meaning the spend per 1000 valid output tokens. Valid output should exclude drafts, self-checks, and changelogs that do not contribute to the final deliverable. Additionally, monitor the reasoning token ratio; a higher ratio indicates more spend is directed toward “internal thinking” rather than “direct output.”
On cloud platforms such as Baidu AI Cloud, you can typically use billing centers, usage dashboards, and monitoring alerts to split consumption by API key, model, or application dimension, and set budget threshold alerts to establish cost baselines for each business line.
3. Constraint Adherence and “Over-Defense”
3.1 Phenomenon: Confirmation One by One, Loops Repeatedly
The prompt provided to the model contained over a thousand words, specifying numerous constraints. The intent was to make the writing more standardized and methodical, ultimately saving time and improving results. However, the model’s behavior was:
- Saying “OK” to the first constraint, then asking whether to proceed to the second.
- If the user replied “pass” or simply “approved” without explicitly stating “all approved,” the model would request further modifications or ask for the second input.
- The third, fourth, and so on followed suit, forming a chain of confirmations.
Since the five tasks were structurally similar and ran simultaneously, this tediousness was not accidental but a consistent behavior. The root cause is the model’s high caution regarding “default consent”: it assumes the user wants to review line by line and only lets it pass all at once when the user explicitly indicates “all approved, no modifications needed.” This caution is intended to prevent misunderstanding the user’s “default” intent, but the actual effect is splitting a task that could be done in one step into several round trips.
3.2 Analysis: The Cost of Over-Defense
- Round Trip Expansion: Each confirmation is a complete model call. With long prompts repeatedly counted, costs grow almost linearly or even super-linearly.
- Context Expansion: Each confirmation round feeds the full previous context back to the model, which is especially severe in long conversations.
- Flow Interruption: Users are forced to make meaningless decisions at each checkpoint, preventing them from focusing on the actual output.
- Ambiguity in Misjudging “Pass”: Words like “pass,” “okay,” “continue,” or “all passed” have different meanings in different contexts, and the model easily interprets them as “partial pass.”
3.3 Mitigation Strategies: Write “Defense” into Constraints
Strategy 1: One-Time Confirmation Protocol. Explicitly declare a “one-time confirmation” rule at the beginning of the prompt, letting the model know the user authorizes direct execution across all constraints rather than asking line by line.
[One-Time Confirmation Protocol]- In this task, all listed constraints are approved by default, no need to ask the user line by line.- If a constraint is ambiguous or conflicting, list it uniformly in the "To Clarify" section at the end of the output, without interrupting the current task.- If the user explicitly replies "All approved" or "No modifications", it is considered authorization for all constraints, proceed directly to execution.
Strategy 2: Hierarchical Constraints. Divide constraints into “hard constraints” (cannot be violated, e.g., format, word count, banned words) and “soft constraints” (negotiable, e.g., tone, style). Clearly inform the model that only hard constraints need line-by-line verification, while soft constraints are at its discretion.
Strategy 3: Hand Off “Checks” to Tools Instead of Conversations. Use a local validation script to automatically check hard metrics like word count, format, and banned words after generation, freeing the model from “self-checking” and reducing unnecessary confirmation rounds.
Strategy 4: Structured Output. Require the model to output in a fixed format (e.g., “Draft + Changelog + To Clarify”), turning the confirmation action from a “conversation” into a “batch review,” allowing the user to read and reply in batches.
3.4 Implications for Developers/Enterprises
Defensiveness itself is not a flaw; it is a byproduct of model alignment and safety design. The problem is that it shifts the cost of “safety” onto the user. Engineering-wise, the “confirmation strategy” should be treated as a configurable parameter rather than forcing every task to endure the default strict mode:
- For internal, low-risk tasks, disable line-by-line confirmation by default.
- For external, formal outputs, retain manual review.
- Use automated validation to replace manual line-by-line confirmation.
4. Writing Quality: “Human Degree” Performance in AI Detection
4.1 Phenomenon: A Near-Perfect “Human Degree” in One Shot
In AI detection tools, three out of the five tasks received a “100% human” judgment, all marked green. This was unimaginable in the past. Previously tested models, even with the highest reasoning tier enabled, only achieved 30%–50% on the best single-shot output, rarely exceeding 60%. The tester set 60% as the threshold, above which the output was considered usable. This time, the single-shot output achieved a near-perfect human degree.
4.2 Analysis: Why the New Model Writes More “Like a Human”
- Writing Pattern Shift: The tester’s judgment is that the new model may have optimized for mainstream AI detection algorithms, resulting in a distinctly different writing style from previous models.
Common Principles of Detection Algorithms: Mainstream AI detection methods generally fall into three categories:
- Perplexity Analysis: Slices text into segments and evaluates the language model’s “surprise” at the text. Higher perplexity means it is more likely human-written.
- Burstiness Analysis: Measures the intensity of sentence length and structure variations. Human writing mixes long and short sentences, while AI text tends to be too uniform.
- Classifier Methods: Trains a binary classification model on labeled data to determine the text source.
The detection algorithm itself is not complex, but it is highly sensitive to “overly regular” text. In the past, testers had to write an internal skill or agent workflow to change periods to commas, shuffle word order, and add sentence structure variations to bring detection scores down.
- Possible Reasons for Writing Pattern Changes: The new model may have adjusted its pre-training data distribution, generation decoding strategy, and explicit alignment with “human writing characteristics.” This is a reasonable guess but requires more samples for verification.
4.3 Risks and Boundaries
- Contingency of Detection Results: Detection tools themselves have false positives. Three “perfect scores” out of five samples might be coincidental, or the tool’s detection logic might have been “bypassed.” It cannot be inferred that the model will perform equally well across all writing tasks.
- Do Not Aim to “Trick the Detector”: Using “human degree” as an optimization target easily slides into “style imitation” rather than “content quality.” True writing quality depends on factual accuracy, logical coherence, information density, and reader value.
- Academic and Compliance Risks: In academic, news, or legal scenarios, evading AI detection may involve academic integrity or platform rule issues. Detection results should be treated as a signal of “whether the style is too templated,” not a “ready-to-deliver” pass.
4.4 Implications for Developers
- Treat “human degree” as a style metric, incorporating it into the evaluation set alongside fact-checking and logic validation.
- Use automated means to detect “templated traces,” such as sentence length variance, vocabulary diversity, and repeated phrase ratios.
- Retain a human polishing step for high-value content; let the model handle the first draft, while humans handle the “human touch” and facts.
5. Engineering Practices: How to Use Stronger Writing Capabilities at Low Cost
5.1 Tiered Routing: Choose Models by Task Difficulty
Not all tasks require the highest tier. You can establish routing strategies by task type:
def route(task: dict) -> str:if task["type"] in ("short_copy", "faq", "summarize"):return "small_model" # Simple tasks, use lightweight modelif task["type"] in ("long_form", "creative", "analysis"):return "frontier_model" # Complex tasks, use flagship modelif task["type"] == "review":return "frontier_model_high" # Review tasks, use high reasoning tierreturn "small_model"
The core idea of the illustrative code above is “use cheap models for simple tasks, flagship models only for complex tasks,” combined with caching and batch processing to reduce unit costs.
5.2 Caching and Batch Processing
- Prefix Caching: Place stable content like system prompts, constraint lists, and task templates at the very beginning of the prompt to leverage platform prefix caching capabilities and reduce repeated input costs.
- Batch Processing: Use batch processing APIs for offline tasks, which typically offer lower pricing.
- Result Caching: Cache output results for identical inputs to avoid repeated generation.
5.3 Monitoring and Alerting
- Split usage by API key, model, and application dimension.
- Set budget threshold alerts to prevent single tasks from spiraling out of control.
- Record the “cost per unit of valid output” for each task as a basis for capacity planning.
5.4 Human-in-the-Loop Design
- For low-risk tasks, disable line-by-line confirmation and switch to “post-generation batch review.”
- For high-risk tasks, retain manual review but template the review checklist to reduce repeated decision-making.
- Use automated validation to replace manual line-by-line confirmation, turning “defense” into a “configurable parameter.”
5.5 Evaluation Set Construction
- Build an evaluation set covering different task types, recording “quality-cost-latency” three-dimensional metrics.
- Run a baseline evaluation for every model upgrade to quantify benefits and costs.
- Include “human degree” detection as one of the style metrics, alongside fact-checking and logic validation.
6. Upgrade Decision Checklist
When deciding whether to upgrade from the previous model to the new model, ask four questions:
- Task Type: Is it simple short text, or long-form creation? The latter is more likely to benefit from the new model.
- Cost Budget: Is the cost per unit of valid output acceptable? Can costs be reduced through routing, caching, and batch processing?
- Interaction Cost: Can round trips be reduced through the “one-time confirmation protocol,” “hierarchical constraints,” and “structured output”?
- Quality Goal: Are you pursuing “human degree,” or factual accuracy, strict logic, and information density? The former is a style metric; the latter is the foundation of writing.
7. Conclusion
The performance of the new generation of models on writing tasks can be summarized in three sentences: writes more like a human, thinks more, defends more tightly. The order-of-magnitude increase in token consumption and the interaction friction caused by constraint confirmations are its most obvious current shortcomings, while the output that is closer to human writing is its most commendable strength.
For developers, the correct approach is neither “switch entirely” nor “dismiss entirely,” but to treat model capabilities as composable resources: use tiered routing to control costs, use constraint protocols to reduce interaction friction, and use evaluation sets to quantify benefits and costs. Only in this way can the new generation of models truly become a productivity tool rather than a new line item on the bill.
