How to Reduce LLM API Costs When Accuracy Is Comparable

    When several models achieve similar accuracy on the same task, their token consumption—and therefore monthly cost—can differ several-fold. This article uses the task of generating structured tables from meeting minutes to show where cost differences come from and presents four actionable optimization methods: prompt simplification, output format constraints, tiered model routing, and monitoring with caching and asynchronous processing. All prices, ratios, and thresholds are illustrative examples and should be re-measured with your own tasks and pricing.

    How to Reduce LLM API Costs When Accuracy Is Comparable

    Large language models have become part of daily operations at many companies, handling office automation, data processing, and report generation. Many teams run a round of testing before launch, find that several models achieve similar accuracy on the same task, and conclude that the choice hardly matters. The real difference usually shows up when the bill arrives. Two models with comparable accuracy can consume several times different amounts of tokens, and the monthly cost gap grows with call volume.

    This article uses the task of “generating structured tables from meeting minutes” to explain where cost differences come from when accuracy is comparable, and provides optimization methods you can apply directly. The prices, ratios, and thresholds in this article are illustrative only; you need to re-measure them with your own tasks and pricing.

    1. Compare by Cost per Correct Result

    When evaluating models, many people only look at public benchmarks or one-off demo results. This information reflects a model’s general capability, but it cannot answer the question “which model is most cost-effective for my task?” The metric you should compare is the average cost per correct result, not just the price per call.

    1.1 Measurement Methodology

    To make evaluation results reproducible, the report should at least document the following:

    1. Samples. Extract real meeting minutes from the business, covering common and edge cases such as colloquial expressions, overlapping multi-speaker dialogue, and mixed dates and amounts. Have business staff annotate the ground truth. A few dozen samples can surface obvious problems; stable conclusions usually require several hundred samples.
    2. Scoring rules. Every rule must be determinable by a program or by a human, such as whether all fields are present, whether values match the original text, whether date formats are consistent, and whether there is extraneous text.
    3. Model versions and parameters. The same model name may behave differently across versions, and parameters such as temperature also affect output. It is recommended to set temperature to 0 to reduce variance, and note this in the report.
    4. Pass rate and sample size. For example: “On 300 samples, judged by the four rules above, the pass rate was 97%.” Writing only “full marks” or “100%” says nothing about whether the test was adequate.

    1.2 The Cost Formula

    Cost per correct result is calculated as follows:

    Cost per correct result = (Input tokens × input price + Output tokens × output price) / Number of correct results

    The following demonstration uses fictional prices. Assume the input price is 1 unit per million tokens and the output price is 4 units per million tokens. Two models perform as follows on 300 samples:

    • Model A: average input 2000 tokens, average output 600 tokens, 291 correct. Total cost = 300 × 2000 × 1 / 1,000,000 + 300 × 600 × 4 / 1,000,000 = 0.6 + 0.72 = 1.32. Cost per correct result ≈ 1.32 / 291 ≈ 0.0045.
    • Model B: average input 2000 tokens, average output 1500 tokens, 293 correct. Total cost = 0.6 + 1.8 = 2.4. Cost per correct result ≈ 2.4 / 293 ≈ 0.0082.

    Both models pass at around 97%, but Model B’s cost per correct result is about 1.8× that of Model A. The entire gap comes from output redundancy.

    2. Where Do Billing Differences Come From?

    Comparable accuracy does not mean identical output. In structured tasks, cost differences usually come from three sources.

    First, output redundancy. Some models like to add greetings, explanations, or summaries before and after the table. For example, when only a block of CSV is needed, the model returns “Sure, here is the table you requested,” wraps it in code block markers, and appends a closing remark. This content has no business value but counts toward output tokens. If each call adds 900 extra output tokens, at the prices above, 300 calls cost roughly 1.08 units more.

    Second, whether reasoning is externalized. Some models perform lengthy internal reasoning before answering. Some APIs return the reasoning content along with the answer, and some also bill it as output tokens. For rule-based tasks like field extraction, this overhead is usually unnecessary. Refer to the platform’s official documentation for the exact billing policy.

    Third, how input is processed. Different models and platforms have different billing and caching rules for long inputs. If the same meeting transcript is sent in full every time, input tokens keep accumulating. If the prompt also contains long, repeated rule descriptions, every call is billed for them again.

    These differences depend on the model, the API, and the task itself. They cannot be judged by intuition—you need to measure them using the methodology above.

    3. Four Actionable Optimization Methods

    The four methods below can be used individually or in combination. Each includes steps, example parameters, applicable boundaries, and troubleshooting points. After any change, go back to the evaluation set from Section 1 and recompute the cost per correct result.

    3.1 Prompt Simplification

    Steps:

    1. Remove duplicated rule descriptions. Keep each rule only once.
    2. Compress field definitions into a table or JSON Schema. For example, use one line—“Date: YYYY-MM-DD, leave blank if missing”—instead of three paragraphs of explanation.
    3. Keep only one or two examples, choosing the samples that best represent edge cases.

    Example parameters: a prompt originally around 2000 tokens is reduced to about 800 tokens. After simplification, use the evaluation set to confirm that accuracy has not dropped significantly; here, “significant” can be defined as more than 1 percentage point below the baseline.

    Applicable boundary: simplification must not remove rules needed to handle edge cases. For example, a rule like “when multiple speakers conflict, use the final confirmed conclusion”—deleting it usually degrades accuracy.

    Troubleshooting: if accuracy drops beyond the threshold after simplification, revert the most recently deleted rule first, restore rules one by one, and identify the one with the largest impact.

    3.2 Output Format Constraints

    Steps:

    1. Use the structured output capabilities provided by the platform, such as JSON mode or a specified schema. If the platform does not support this, explicitly require in the prompt: “Output CSV only. No code blocks, no explanations, no greetings.”
    2. Perform parse validation on the application side. Retry at most once on parse failure, and log the failure reason.
    3. Set a max_tokens limit. Example: if a transcript typically generates a 400-token table, set the limit to around 600—about 1.5× the expected value.

    Applicable boundary: setting the limit too low will truncate long tables and cause missing fields. First measure output length on long transcripts, then determine the limit.

    Troubleshooting: if the output still contains greetings, first check whether the prompt was loaded correctly, then check whether post-processing logic strips extra content. If the parse failure rate exceeds 2%, the format requirements do not match actual output—loosen the schema or adjust the examples.

    3.3 Tiered Model Routing

    The idea is to let simple samples go to a cheaper model and send only difficult samples to a stronger model.

    Steps:

    1. Extract difficulty features from the evaluation set, such as transcript length, number of fields, and whether it contains tables or multi-speaker dialogue.
    2. Define routing rules with these features. For example, samples with fewer than 8 fields and transcripts under 1500 characters are first handled by the smaller model.
    3. The smaller model’s output is validated against the scoring rules. Samples that fail validation are escalated to the larger model for reprocessing.
    4. Randomly sample about 10% of the smaller model’s results for manual review, and continuously monitor misjudgments.

    Example parameters: if the escalation rate after routing is 25%, most calls only incur the smaller model’s cost.

    Applicable boundary: the router itself has costs and misjudgment risks. Routing rules must be validated on the evaluation set—do not set thresholds on intuition alone.

    Troubleshooting: if the escalation rate exceeds 40%, the threshold is too strict, or the smaller model is simply unsuitable for the task. Re-examine feature selection, or directly reduce the routing ratio.

    3.4 Monitoring, Caching, and Asynchronous Processing

    Monitoring is the prerequisite; caching and asynchronous processing are the two main levers for reducing cost.

    Monitoring. It is recommended to track these metrics daily: cost per correct result, pass rate, average input and output tokens, and escalation rate. The alert threshold can be set at daily cost exceeding 30% of the 7-day average. The threshold is only an example—adjust it to your business’s volatility.

    Caching. A recommended cache key:

    Cache key = hash(model name + model version + prompt version + transcript content hash + temperature)

    Do not include random fields such as timestamps or request IDs in the cache key, or the hit rate will approach zero. The TTL can be set to 7 days. When a transcript is modified or the prompt version changes, the relevant cache entries should be invalidated immediately.

    Note that the cache stores historical results. If a transcript has been modified but the cache has not been invalidated, users will receive stale tables. Therefore, the invalidation strategy must be bound to content versions, not just the TTL.

    Asynchronous processing. For tasks that do not require real-time responses—such as batch-generating weekly report tables overnight—place them in a queue for unified processing. Use exponential backoff for retries, up to 3 attempts. Each task should carry an idempotency key to prevent duplicate writes of the same table during retries.

    There are three main consistency risks with asynchronous processing. First, before a task completes, users still see old data. Second, retries may cause duplicate writes, which must be handled with idempotency keys. Third, when the queue backs up, rate limiting is needed, otherwise it will squeeze resources from other tasks.

    Troubleshooting: if the cache hit rate is below 10%, first check whether the cache key contains random fields. If daily cost persistently exceeds the threshold, check for retry storms or non-deduplicated batch tasks.

    4. Pre-Launch Checklist

    Before going live, verify each of the following:

    • The evaluation set has at least 300 samples and includes edge cases.
    • Scoring rules can be stably determined by a program or a human.
    • Model version, temperature, and prompt version are recorded.
    • Cost per correct result has been computed and compared against the baseline.
    • Output format passes parse validation, with retries and logging on failure.
    • A max_tokens limit is set and verified against long transcripts to ensure no truncation.
    • Routing rules have been validated on the evaluation set, and the escalation rate is acceptable.
    • The cache key contains no random fields, and the invalidation strategy is bound to content versions.
    • Asynchronous tasks have idempotency keys, retry limits, and rate limiting.
    • Monitoring and alerting are in place, with an owner responsible for responding.

    5. Risk Notes

    Pricing and billing rules change over time. All figures should be based on the official pricing page and the latest documentation. The prices, thresholds, and ratios in this article are illustrative examples.

    Meeting minutes typically contain sensitive information about personnel, customers, and finances. Before sending transcripts to an external model, confirm compliance with your company’s data governance requirements. Caches and logs should not store full transcript content—apply masking and set access permissions.

    The goal of cost reduction is to lower expenses while preserving result quality. If saving money sacrifices the accuracy of critical fields, the cost per correct result will actually rise. Therefore, every optimization should be validated on the evaluation set before deciding whether to keep it.