How to Evaluate LLM APIs for Cost, Quality, and Business Fit

    Choosing an LLM API requires more than comparing token prices. This guide explains how to calculate the real cost per successful task, build business-specific evaluation datasets, and compare models using accuracy, formatting compliance, refusal quality, and hallucination rates. It also includes Python examples for cost estimation and API-based testing.

    1. Calculate the Real Cost First

    Most LLM APIs charge separately for input and output tokens. The list price only tells you the cost per token; it does not show how much a business task will actually cost. Three factors have the greatest impact:

    • Token usage per task
    • Task success rate
    • The number of retries after failures

    Output tokens are usually more expensive than input tokens, so prompt design and output-length control can significantly affect costs. Some providers also offer discounts for cached prompts or batch requests, but the eligibility requirements vary. Always verify these details in the official documentation instead of estimating from memory.

    You can estimate the cost of one successful task with the following formula:

    1. Cost per successful task =
    2. (input tokens × input price + output tokens × output price) ÷ task success rate

    Consider this hypothetical example:

    • Model A charges $2 per million input tokens and $8 per million output tokens. Each task uses an average of 3,000 input tokens and 1,500 output tokens, with a 60% success rate.
    • Model B charges $4 per million input tokens and $16 per million output tokens. Each task uses an average of 1,200 input tokens and 500 output tokens, with a 90% success rate.

    The estimated costs are:

    • Model A: approximately $0.030 per successful task
    • Model B: approximately $0.014 per successful task

    The model with the higher token prices has the lower cost per successful task. This demonstrates why API selection must consider pricing, token usage, and success rate together.

    1. def cost_per_success(
    2. input_tokens,
    3. output_tokens,
    4. in_price,
    5. out_price,
    6. success_rate,
    7. ):
    8. """Prices are expressed per million tokens."""
    9. if not 0 < success_rate <= 1:
    10. raise ValueError("success_rate must be in the range (0, 1].")
    11. raw_cost = (
    12. input_tokens * in_price + output_tokens * out_price
    13. ) / 1_000_000
    14. return raw_cost / success_rate

    Average token usage should come from production logs rather than manual estimates. Changes to prompt templates and shifts in user-input length can both alter the average significantly.

    2. Use Business Samples Instead of Generic Leaderboards

    Public benchmarks measure model performance on standardized datasets. Their results may differ substantially from the data distribution and requirements of a specific business. A model that ranks highly on a general leaderboard may not be the best choice for tasks such as support-ticket classification or contract-field extraction.

    A more reliable approach is to build an evaluation set based on your own use case.

    1. Extract 200 to 500 samples from real logs, covering both common scenarios and edge cases.
    2. Define a verifiable reference answer or scoring rule for every sample.
    3. Score each model across relevant dimensions, such as factual accuracy, format compliance, appropriate refusal behavior, and hallucination rate.
    4. Keep evaluation parameters fixed, including sampling temperature, maximum output length, and the system prompt, so that results remain comparable across models.

    Evaluation can use rule-based matching or a second model as an evaluator. When using model-based evaluation, manually review a sample of the results to verify that the evaluator generally agrees with human judgment.

    The following example uses a hypothetical API endpoint and response format. Adapt the request details to the documentation of the API provider you use.

    1. import json
    2. import os
    3. import urllib.request
    4. BASE_URL = os.environ["LLM_BASE_URL"] # For example: https://api.example.com/v1
    5. API_KEY = os.environ["LLM_API_KEY"] # Read the key from an environment variable
    6. def call_model(model: str, prompt: str) -> str:
    7. body = json.dumps({
    8. "model": model,
    9. "messages": [{"role": "user", "content": prompt}],
    10. "temperature": 0,
    11. }).encode("utf-8")
    12. request = urllib.request.Request(
    13. f"{BASE_URL}/chat",
    14. data=body,
    15. method="POST",
    16. headers={
    17. "Content-Type": "application/json",
    18. "Authorization": f"Bearer {API_KEY}",
    19. },
    20. )
    21. with urllib.request.urlopen(request, timeout=60) as response:
    22. data = json.loads(response.read().decode("utf-8"))
    23. return data["choices"][0]["message"]["content"]
    24. def is_valid_json(text: str) -> bool:
    25. try:
    26. json.loads(text)
    27. return True
    28. except json.JSONDecodeError:
    29. return False
    30. def evaluate(model: str, samples: list[dict]) -> dict:
    31. json_total = 0
    32. json_ok = 0
    33. correct = 0
    34. for sample in samples:
    35. output = call_model(model, sample["prompt"])
    36. if sample.get("expect_json"):
    37. json_total += 1
    38. json_ok += is_valid_json(output)
    39. correct += bool(sample["judge"](output))
    40. return {
    41. "accuracy": correct / len(samples),
    42. "json_format_rate": (
    43. json_ok / json_total if json_total else None
    44. ),
    45. }
    46. }