How to Evaluate LLM Performance and Cost in Office and Financial Scenarios
Deploying large language models in office and financial workflows requires more than leaderboard scores. This article explains why public benchmarks fall short for real business decisions, and walks through how to build a reliable evaluation process: constructing task sets from real workflows, running blind tests, measuring answer reliability and hallucination rates, interpreting composite and coding benchmarks, and calculating true per-task cost including retries, human review, and rework. A sample Python snippet shows how to aggregate evaluation results into key metrics, and the article closes with practical rollout recommendations covering phased deployment, human oversight, permission control, and continuous re-evaluation.
Once large language models (LLMs) enter enterprise office and financial workflows, the question teams ask most often is: which model is better? It sounds simple, but no single score can answer it. Leaderboard results come from specific question sets, and task construction, scoring rules, and runtime environments all affect the outcome. Applying those scores directly to your own business often leads to badly skewed conclusions. Starting from evaluation design, this article explains how to build a trustworthy model selection process for office and financial scenarios.
1. The Limits of Leaderboard Scores
Composite leaderboards compress multiple capabilities into a single number. That compression makes comparison easy but loses information. First, difficulty distributions differ across question sets—a math problem and a contract clause review demand almost unrelated capabilities. Second, some evaluations let models choose their own output format, and format differences affect automated scoring. Third, leaderboards are largely built on public data, which model providers may have optimized against, creating a gap between scores and real business performance.
Leaderboards are therefore useful for narrowing the candidate pool, not for making procurement or go-live decisions. The real basis for a decision should come from tasks that closely resemble your own business.
2. Designing Evaluations for Office and Financial Scenarios
The quality of the evaluation set determines how credible the conclusions are. A recommended construction process:
- Extract tasks from real tickets, reports, emails, and policy documents. Keep the original context and complete de-identification.
- Stratify by task type—for example, information extraction, summarization, data reconciliation, report writing, compliance Q&A, and multi-step workflow handling. Define clear success criteria for each layer.
- Write reference answers or scoring rubrics for every task. The rubric should specify which errors are fatal, such as incorrect amounts, dates, or clause citations.
- Make sure the sample size covers long-tail cases. With too few samples of a single task type, conclusions are easily swayed by individual examples.
Financial scenarios especially require separating “fluent wording” from “factual correctness.” A professional-sounding response can still be wrong on key figures. Score factual accuracy and expression quality separately.
3. Blind Testing and Scoring Consistency
Blind testing reduces reviewers’ subjective bias. In practice: hide model names, randomize output order, and have reviewers score independently. To judge whether the results are reliable, also pay attention to the following:
- Reviewer composition: Business teams and risk control teams see things differently. A mixed panel better reflects real usage.
- Inter-rater agreement: Measure it with Cohen’s kappa or Krippendorff’s alpha. If agreement is low, revise the scoring rubric first, then re-run the review.
- Confidence intervals: Report standard deviations or confidence intervals alongside means. When two models’ means differ by only a fraction of a point, there is usually no meaningful gap.
4. Knowledge Reliability: What Should the Model Do When It Doesn’t Know?
In business settings, a model that “forces an answer” is more dangerous than one that answers incorrectly, because wrong answers often look convincing. To evaluate knowledge reliability, design three types of questions:
- Verifiable factual questions: The answer can be found in the provided documents. These test accuracy.
- Unanswerable questions: The documents contain no relevant information. The correct behavior is to state that the answer cannot be determined. These test refusal ability.
- Questions with misleading premises: The question itself contains a false assumption. These test whether the model follows the premise blindly.
Track three proportions separately: “answered correctly,” “correctly refused,” and “incorrectly forced an answer.” The forced-answer rate is one of the most important metrics to watch before go-live. Common ways to reduce it include requiring the model to cite source passages, explicitly permitting “cannot be determined” in the prompt, and adding post-hoc validation for critical fields such as amounts and dates.
5. How to Read Composite Indices and Coding Benchmarks
A composite index reflects a model’s average performance across multiple tasks and is suitable for coarse filtering. Coding benchmarks are closer to engineering teams’ needs, but verify whether the problems resemble a real codebase. When reading this kind of data, ask: Is the question source public? Does it include multi-file modifications? Do the test cases cover boundary conditions? Is multi-round debugging allowed?
If two models’ composite scores are close and your target scenario depends on a specific capability, defer to that capability’s performance.
6. Cost Accounting: Calculate per Task
Model pricing is usually based on input and output token counts, but business cost also includes retries, human review, and rework on failed tasks. Per-task cost can be estimated as:
Per-task cost = (input tokens × input unit price + output tokens × output unit price) × average number of calls + human review cost × review ratio + rework cost × failure rate
Two points deserve attention in this formula. First, the average number of calls changes with prompt design and the tool-calling chain. Second, the failure rate must come from real evaluation—it cannot be replaced with an assumed value. Two models may have similar per-call prices, but if their failure rates differ by a factor of two, the final costs can diverge significantly.
7. Sample Code: Aggregating Evaluation Results
The following code demonstrates how to aggregate evaluation records into several key metrics. The data structure is illustrative only; actual fields should be adjusted to the business.
from dataclasses import dataclass@dataclassclass EvalRecord:correct: bool # answer is correctrefused: bool # correctly refused an unanswerable questionhallucinated: bool # forced an answer on an unanswerable questioncost_usd: float # per-task costneeds_review: bool # whether human review is neededdef summarize(records: list[EvalRecord]) -> dict:n = len(records)if n == 0:raise ValueError("evaluation records are empty")return {"accuracy": sum(r.correct for r in records) / n,"forced_answer_rate": sum(r.hallucinated for r in records) / n,"avg_cost_per_task": sum(r.cost_usd for r in records) / n,"review_ratio": sum(r.needs_review for r in records) / n,}
Aggregated results should be stored together with the raw evaluation logs so they can be reproduced and audited later.
8. Rollout Recommendations
A three-step approach is recommended before go-live. Step one: a limited grayscale release covering only low-risk tasks, with full input and output logging retained. Step two: add human review for high-risk steps, such as amounts, contract clauses, and externally disclosed content. Step three: establish a continuous evaluation mechanism—re-run the benchmark set after every model version update or prompt change.
Permissions and data security matter just as much. Model access should follow the principle of least privilege, and sensitive data should be de-identified or classified before entering prompts. Model outputs should not be written directly into core systems; they must pass through a validation step first.
The key to model selection is not finding the highest-scoring model, but finding the option that performs most consistently under your own task, risk, and budget constraints. The closer the evaluation process is to real business, the more reliable the conclusions.
