Short Chain-of-Thought, Higher Translation Quality: A Practical Evaluation Method for Business Letters
Business letters carry legal and commercial weight, where a single mistranslated phrase can shift a party's stance, distort facts, or weaken a commitment—so "does it read like English" is the wrong quality metric. This article presents a practical method for evaluating Chinese-to-English business letter translation with a seven-dimensional anchored scorecard executed by an LLM-as-a-judge. It explains how to design a test letter that embeds wording-strength markers and verifiable hard constraints, how to plant a deliberately misleading draft as a distractor, how to turn the scorecard into an executable judge prompt, and how to build a robust pipeline with concurrent calls, idempotent persistence, and raw-output retention. Test results show the slowest models did not earn a matching quality return, indicating reasoning time is not a reliable proxy for translation quality. It closes with the method's limitations and a path toward dynamic scorecards and automated quality gating.
1. Why Business Letter Translation Deserves Its Own Evaluation
1.1 A High-Stakes Scenario: A Business Letter Is a Legal Document With a Position
Business letters are the formal communication channel for companies during contract performance and commercial negotiation. In scenarios involving delivery delays, claims for liquidated damages, quality objections, compensation, and settlement, the wording itself is part of the legal effect. When translating such text between Chinese and English, common errors—mistranslated low-frequency terms, dropped sentences, and shifts in tone—can directly change the legal and commercial meaning of the letter.
The risks of business letter translation fall into three categories:
- Stance drift: rendering “cannot agree” as a phrasing that leaves room for negotiation effectively concedes ground in a dispute.
- Factual distortion: errors in delivery dates, amounts, penalty percentages, or contract clause references may create statements that work against your own side.
- Effect mismatch: expressions like “尽量” (do our best) and “须” (must) correspond in Chinese to a “best-efforts obligation” and a “mandatory obligation” respectively; using the wrong strength changes how binding a commitment is.
For this reason, the core metric of business letter translation is not “does it read like English,” but “have the stance, facts, and boundaries of commitments been preserved without distortion.”
1.2 Blind Spots in General-Purpose Evaluation
Common automated metrics for measuring machine translation, such as n-gram-overlap-based BLEU and COMET (oriented toward semantic representation), mainly answer “how close is the translation to a reference translation.” They work for general text but have clear blind spots in the business letter scenario:
- There is only one reference wording, yet both “we regret that we must decline” and “we are unable to agree” may be acceptable renderings of a polite refusal; literal comparison would reach the opposite conclusion.
- “尽量压缩工期” (compress the schedule as much as possible) and “确保压缩工期” (ensure the schedule is compressed) differ very little in word overlap, yet their legal meanings are worlds apart.
- Metrics typically struggle to cover constraints such as terminology consistency, letter-writing conventions, and output format.
For this kind of task, a human-designed scorecard is needed to break “expert judgment” into anchorable, reviewable dimensions, and then a large model is used as a judge to execute at scale. This “LLM-as-a-judge” approach can significantly reduce cost and improve consistency.
2. Test Design: A Business Letter That Neither Shows Weakness nor Breaks Relations
This test selected a Chinese-to-English business letter in a “delivery delay + liquidated damages dispute” scenario, had multiple large models answer the same question, and had a judge model score them across seven dimensions. The core idea of the task design is a single sentence: a refusal is a refusal, “do our best” is not a guarantee, and the boundary of a commitment does not move an inch.
2.1 Wording Strength Is Stance
The source text presumes that our side has confirmed the fact of the delay but is willing to bear only limited remedial responsibility, refuses to guarantee an earlier delivery date, and reserves its position on the counterparty’s liquidated damages claim. The Chinese original embeds a whole row of strength markers, each corresponding to a specific legal and commercial meaning:
| Chinese expression | Strength attribute | Acceptable translation | Distorted translation | Consequence |
|---|---|---|---|---|
| 须先行说明 | Mandatory obligation | must be stated in advance | should be noted | Mandatory obligation weakened to a suggestion |
| 恕难同意 | Polite but clear refusal | we regret that we must decline | we cannot fully agree | Becomes partial agreement, leaving room for haggling |
| 只能承诺十二月二日,无法保证更早 | Commitment boundary | we can only commit to December 2 and cannot guarantee an earlier date | we may deliver around December 2 | A firm commitment softened into a vague expectation |
| 尽量压缩工期 | Best-efforts obligation | we will do our best to shorten the schedule | we will ensure the schedule is shortened | Best-efforts obligation strengthened into a result guarantee |
| 顺颂商祺 | Chinese letter closing courtesy | Handled per English letter conventions | Literal translation | Breaks the stylistic norms of English business letters |
The “distorted translations” in the table do not violate grammar; some even read as more “native.” This is precisely the trap built into the task: fluent on the surface, but substantively changing the stance.
2.2 Hard Constraints: Bottom-Line Requirements That Can Be Verified Item by Item
Beyond wording strength, the task set several hard constraints that can be verified one by one:
- Clause references: “Article 7, Clause 2” must map accurately to the hierarchical expression used in English contracts (e.g., Article 7, Clause 2), with no level errors or omissions.
- Dates and amounts: all dates and amounts must exactly match the source, not be rewritten as relative time or approximations.
- Ratio conversion: “千分之三” (three per thousand) must be converted to 0.3% and aligned with the “0.3%” used in the counterparty’s letter; it must not be written as “three per thousand,” which creates ambiguity.
- Reuse of existing wording: the counterparty’s letter already contains two English expressions, “liquidated damages at 0.3% per day” and “acceptance criteria”; the reply must reuse them verbatim and must not rewrite them as “penalty” or other synonyms.
- Letter conventions: Chinese courtesies such as “顺颂商祺” must be handled per English letter conventions, not translated literally.
- Clean output: output only the translation itself, without added explanations, notes, or reasoning.
Among these, “千分之三 = 0.3%” is a common numerical trap in Chinese business writing. The source uses a fractional expression while the counterparty’s letter uses a percentage; the translator must complete the conversion and align with the counterparty’s wording. Reusing existing wording is to maintain consistency of phrasing across the two sides’ correspondence.
2.3 Distractor: A Colleague’s Draft That “Must Not Be Referenced”
The task included a colleague’s draft translation as a “preliminary version for reference.” This draft intentionally planted several strength distortions:
- a polite refusal weakened to “we cannot fully agree”;
- “尽量压缩工期” strengthened to “we will ensure the schedule is shortened”;
- “只能承诺十二月二日” softened to “we may deliver around December 2”;
- one purpose adverbial dropped entirely.
This distractor is highly deceptive: complete in format, fluent in language, and carrying a low-risk label of “for reference only.” If a model follows the draft’s wording, its score on the tone-strength dimension collapses immediately. Its design intent is to distinguish “independent judgment” from “referencing existing material,” testing whether the model can hold to the original task constraints in the presence of misleading context.
3. Scoring System: A Seven-Dimension Anchored Scorecard
3.1 Why Anchors Are Needed
The most common bias in LLM judges is the “overall impression score”: giving high marks to a fluent translation and dragging down all dimensions when they see a few awkward expressions. The way to suppress this bias is to have the judge only answer “does it satisfy a given observable criterion.” This evaluation breaks scoring into seven dimensions, each scored 0–1, and explicitly requires the judge to cite source-text fragments as evidence.
The seven dimensions and example anchors are as follows:
- Semantic accuracy: whether factual information (dates, amounts, clauses) is mistranslated, added, or dropped.
- Tone strength: whether the strength of obligation, refusal, and commitment matches the source.
- Completeness and coherence: whether any sentences are dropped and whether logical relations (cause-effect, contrast, concession) are preserved.
- Terminology and existing-wording consistency: whether contract clauses, penalty ratios, and acceptance criteria follow the agreed expressions.
- Business letter conventions: whether salutations, paragraphing, and closings conform to English letter conventions.
- Fluency: whether the English is natural and professional.
- Output format: whether it contains only the translation, with no leaked reasoning or extra explanation.
Example anchors:
- Semantic accuracy
- 1 point: all dates, amounts, ratios, and clause references map item by item to the source.
- 0 points: facts that contradict the source appear, or information is dropped or substantially added without authorization.
- Intermediate: minor deviations that do not affect core facts (e.g., articles, singular/plural).
- Tone strength
- 1 point: “须” is rendered as a strong must-type obligation, the polite refusal is clear, and the “can only commit… cannot guarantee” boundary is fully preserved.
- 0 points: any instance of weakened refusal, strengthened obligation, or softened commitment.
- Terminology and existing-wording consistency
- 1 point: “liquidated damages at 0.3% per day” and “acceptance criteria” are used verbatim, and “千分之三” accurately maps to 0.3%.
- 0 points: rewritten as synonyms, or the ratio conversion is wrong.
- Business letter conventions
- 1 point: courtesies are handled per English letter conventions, and the overall format fits a formal business letter.
- 0 points: courtesies are translated literally, or the format clearly departs from business letter conventions.
The total score defaults to the arithmetic mean of the seven dimensions; if the business values stance more highly, the semantic and tone dimensions can be weighted.
3.2 From Anchors to the Judge Prompt
The scorecard needs to be turned into instructions the judge model can execute. The prompt should contain at least four parts:
- Role and goal: define the judge’s identity and the task goal of “reviewing business letter translation quality.”
- Scoring rules: give anchors dimension by dimension and specify a 0–1 continuous score range.
- Evidence requirement: each dimension must cite corresponding fragments from the source and the translation to justify point deductions or awards.
- Output constraints: output JSON in a specified format for easy program parsing and aggregation.
A sample prompt is as follows:
You are a rigorous reviewer of business letter translation quality. Based on thescorecard below, score the [Chinese source] and the [candidate English translation]dimension by dimension (0–1, up to two decimal places), and for each dimension citethe corresponding fragments from the source and the translation as evidence.Scoring dimensions and anchors:1. Semantic accuracy: whether factual information is mistranslated, added, or dropped.2. Tone strength: whether the strength of obligation, refusal, and commitment matchesthe source; a refusal must not be weakened, and a best-efforts obligation must not bestrengthened into a result guarantee.3. Completeness and coherence: whether any sentences are dropped and whether logicalrelations are preserved.4. Terminology and existing-wording consistency: must reuse "liquidated damages at 0.3% per day"and "acceptance criteria"; "千分之三" must map to 0.3%.5. Business letter conventions: courtesies must be handled per English letter conventions,not translated literally.6. Fluency: whether the English is natural and professional.7. Output format: only the translation, with no explanation or reasoning.Rules:- Judge only from the source and translation; do not reference any draft or reference version;- Each dimension must provide an evidence fragment; if no evidence can be found, score 0;- Output strictly as the following JSON and nothing else:{"scores": {"semantic": 0.0, "tone": 0.0, "completeness": 0.0,"terminology": 0.0, "convention": 0.0, "fluency": 0.0, "format": 0.0},"evidence": {"semantic": "...", "tone": "...", "...": "..."},"overall": 0.0, "comment": "..."}
3.3 Four Design Points to Constrain the Judge’s “Free Play”
- Judge independence: the judge is a model that did not participate in answering, avoiding the bias of “judging yourself.”
- Mandatory evidence: judgments without a source-text fragment are considered invalid, turning subjective impressions into verifiable evidence.
- Per-dimension independence: each dimension is scored separately, preventing a fluent translation from “sneaking through” on the tone dimension.
- Auxiliary verification: sample-check the judge’s own stability, e.g., rescoring the same input to observe variance, and manually review a portion of samples.
4. Engineering Implementation: Concurrent Calls, Idempotent Persistence, and Raw-Output Retention
The credibility of an evaluation depends not only on the task and scorecard but also on the engineering implementation. The execution environment for this test was: a single-day batch run, Python 3.11, model calls made via a general HTTP client, and all models called concurrently through the same compatible interface. The specifics are as follows:
- Input alignment: all models receive exactly the same prompt and source text.
- Result retention: parse and save the content of the first message returned by the model, without any cleaning of the raw text.
- Idempotent persistence: check whether a result already exists by filename, supporting resumption from checkpoints.
- Evidence trail: save elapsed time and token usage for later analysis.
4.1 Idempotent Persistence and Resumption
If a batch evaluation is interrupted during the judge scoring stage or by network fluctuation, re-calling completed tasks wastes time and quota, and different batches are also affected by server load, undermining cross-model comparability. The solution is to use the “task filename” as the idempotency key:
import jsonimport timeimport urllib.requestfrom concurrent.futures import ThreadPoolExecutorfrom pathlib import PathAPI_URL = "https://<compatible HTTP endpoint>/chat/completions"API_KEY = "<configure the key in an environment variable>"MODEL_NAMES = ["model_a", "model_b", "..."] # model identifiers are provided by the vendorINPUT_PATH = "input/letter_zh.txt"OUTPUT_DIR = Path("outputs")RETRY = 2def call_model(model: str, letter: str) -> dict:"""Call the compatible HTTP endpoint and retry a limited number of times on 5xx errors."""payload = {"model": model,"messages": [{"role": "user", "content": f"Please translate the following business letter into English:\n{letter}"}],"stream": False,}req = urllib.request.Request(API_URL,data=json.dumps(payload, ensure_ascii=False).encode("utf-8"),headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},method="POST",)last_err = Nonefor attempt in range(RETRY + 1):try:with urllib.request.urlopen(req, timeout=300) as resp:return json.loads(resp.read().decode("utf-8"))except urllib.error.HTTPError as e:last_err = eif e.code < 500 or attempt == RETRY:raise # deterministic errors such as 4xx, or retries exhausted, are raised directlytime.sleep(2 ** attempt) # exponential backoffraise last_errdef run_one(model: str, letter: str) -> None:dst = OUTPUT_DIR / f"{model}.json"if dst.exists(): # idempotent: skip if a result already existsprint(f"[skip] {model}")returnstart = time.perf_counter()try:data = call_model(model, letter)# keep the raw text, no cleaning: format-dimension deductions need the raw output as evidencecontent = data["choices"][0]["message"]["content"]record = {"model": model,"elapsed_s": round(time.perf_counter() - start, 2),"usage": data.get("usage"),"raw": content,}except Exception as e:record = {"model": model, "error": repr(e),"elapsed_s": round(time.perf_counter() - start, 2)}tmp = dst.with_suffix(".tmp")tmp.write_text(json.dumps(record, ensure_ascii=False, indent=2), encoding="utf-8")tmp.replace(dst) # atomic write, avoids half-written files on interruptiondef main() -> None:OUTPUT_DIR.mkdir(parents=True, exist_ok=True)letter = Path(INPUT_PATH).read_text(encoding="utf-8")with ThreadPoolExecutor(max_workers=4) as pool:# one model's failure doesn't affect other tasks; map returns results in input orderlist(pool.map(lambda m: run_one(m, letter), MODEL_NAMES))if __name__ == "__main__":main()
This code contains four engineering points:
- Idempotent skip: when rerunning the script, existing files are skipped directly, without consuming more time and quota.
- Atomic persistence: write a temporary file first, then replace it as a whole, so a process interruption does not leave a partial file that could be mistaken for “complete.”
- Raw retention: raw output is not cleaned in advance, so point deductions on the format dimension have evidence to reference.
- Failure isolation: one model’s failure does not affect other tasks; error types and elapsed time are also persisted, making it easy to diagnose afterward whether it was a timeout, rate limiting, or content interception.
For higher concurrency, you can use an async client with a semaphore to control the concurrency level. The concurrency ceiling should be based on the rate-limit responses returned by the provider (usually a 429 status code), and such responses should have their own backoff retry.
4.2 Engineering Requirements for the Judge Stage
- Same interface: candidate translations are submitted to the judge through the same entry point, carrying the raw output.
- Traceability: judge output is likewise persisted by file, recording the judge identifier and prompt version number used.
- Verifiability: samples that fail JSON parsing are retried separately; when unrecoverable, they are recorded as invalid samples and excluded from the mean calculation.
5. Results and Interpretation: Thinking Longer Does Not Equal Translating Better
5.1 Key Data
| Model | End-to-end time (s) | Seven-dimension mean |
|---|---|---|
| Model C | 179.29 | 0.64 |
| Model H | 193.93 | 0.46 |
These two data points decouple “reasoning time” from “translation quality” clearly: of the two slowest models, one scored in the middle and the other came dead last.
It should be emphasized that this set of data only corresponds to the performance of the online versions on the day of collection, not a long-term evaluation of any model. After vendors silently update weights, the scores will drift, so reproduction requires running another round of comparison yourself.
5.2 Why Thinking Longer May Actually Be Worse
From the task’s characteristics, at least three mechanisms can explain this phenomenon:
- Overthinking introduces wavering: models that require long-chain reasoning may repeatedly weigh multiple candidate translations and ultimately deviate from the initial judgment that “a refusal is a refusal.”
- Reference-material inducement: facing a “for reference only” draft, a model may treat the distorted expressions in the draft as reasonable options and be led in the wrong direction during reasoning.
- Output-format breakdown: the longer the reasoning process, the more likely issues such as leaked reasoning content and mixed prefixes/suffixes appear, directly dragging down the format dimension score.
More generally, business letter translation tests constraint-following and style consistency, not complex reasoning ability. There is no necessary positive correlation between time and quality.
5.3 The Applicability Boundary of the Conclusions
Nine models, one task, single-day single collection—this is a small-scale smoke test, suitable for finding systemic problems, but not enough to support a long-term conclusion of “who is stronger.” If it is to be used for selection, it is recommended to at least:
- Expand the tasks: cover multiple letter types such as claims, reminders, settlements, and complaint responses.
- Repeat sampling: run each model multiple times to assess the variance of outputs.
- Manual spot checks: sample-check whether judge scores match human judgment.
6. Limitations, Risks, and Future Evolution
6.1 Limitations of the Current Method
- Judge preference: the model judge may have tendencies such as preferring verbosity or preferring a particular style, which need to be monitored through manual spot checks.
- Sample size: single-task, single-run results are heavily affected by random factors, giving scores a wide confidence interval.
- Version drift: after online services are silently updated, historical scores are no longer comparable, so a version baseline needs to be established.
6.2 Dynamic Scorecards and Quality Gating
The scorecard is not set in stone. New failure patterns (e.g., new courtesies, new terminology conflicts) can be extracted from real business translation-rework records, anchors can be continuously supplemented, and small-sample regression validation can be done on the scorecard.
The evaluation pipeline can also be embedded into the content production process to form automated quality gating:
- Style guide or terminology base → constrained generation
- Translation → rule validation (dates, amounts, clause numbers, banned expressions)
- Model review (per-dimension scoring)
- Low-score alert → manual review → rework records flow back into the scorecard
Among these, rule validation is suited to constraints with strong determinism, such as comparing numbers and clause references; semantic and tone judgment still relies on review. The two complement each other and can reduce missed detections.
7. Conclusion
The value of business letter translation evaluation lies not in crowning a “champion model,” but in turning vague expert judgment into a reproducible, auditable engineering process: designing tasks and distractors with the idea that “wording strength is stance,” constraining the judge with a seven-dimensional anchored scorecard, and ensuring result stability through concurrent calls and idempotent persistence. In testing, two large models that each took nearly three minutes failed to earn a corresponding return, which shows that time is not a valid indicator of translation quality.
In real business, such tasks often require a domain glossary, style guide, and enterprise knowledge working together. By connecting model review to the business process through Baidu AI Cloud’s large model platform, enterprises can continuously monitor the quality baseline of business letter translation and reduce the stance and compliance risks in external communication.
