1. Introduction: Translating the Capital Narrative into Engineering Metrics

Funding and valuation headlines for large models have kept climbing, and round size, post-money valuation, and equity dilution are often lumped together. For developers and architects, the more useful question is: what ultimately supports the growth? For an AI service, the answer usually comes down to unit economics — how much it costs to complete one call, process one million tokens, or serve one active user, and how much revenue that produces.

This article translates the capital narrative into engineering language. It separates valuation from money raised and from dilution, breaks down the cost structure of AI inference, and lays out practical optimization techniques, sample code, and a metrics system to help teams find a sustainable balance among cost, quality, and capacity.

2. Valuation, Round Size, and Dilution: Get the Definitions Right First

2.1 Valuation is not cash, and money raised is not profit

Valuation describes what the whole company is worth. The amount raised depends on how many new shares the company issued. The relationship is:

New share percentage = Amount raised ÷ Post-money valuation
Pre-money valuation  = Post-money valuation − Amount raised

If public figures are quoted on a pre-money basis, convert them as follows:

New share percentage = Amount raised ÷ (Pre-money valuation + Amount raised)

To demonstrate the calculation, the table below uses a set of hypothetical data points. They do not refer to any real funding event:

Hypothetical data point	Amount raised (USD 100M)	Post-money valuation (USD 100M)	New share percentage
Point A	7.5	184	~4.1%
Point B	35	615	~5.7%
Point C	130	1830	~7.1%
Point D	300	3800	~7.9%
Point E	650	9650	~6.7%

Two things stand out.

First, a higher valuation does not imply a proportional transfer of ownership. Point E raised far more than Point A, yet the new share percentage did not scale up at the same rate.

Second, the amount raised is not wealth on paper; it is cash that has to be spent in the future. A substantial share turns into long-term commitments: accelerators, data centers, power, R&D, and bandwidth. If revenue growth on the demand side fails to keep up, that spending shows up on the engineering cost side as depreciation, low utilization of idle resources, and a high cost per call.

2.2 Price anchors hide the real problem

A run of large rounds turns price itself into a new reference point: if someone was willing to invest last round, a higher price next round seems self-explanatory. The same thing happens in everyday pricing. What an API is worth is often anchored to what leading providers charge; how much faster one accelerator runs a given model is often anchored to the previous generation of hardware.

Anchors help communication, but they easily replace the question that should actually be asked: is there cost and quality behind the anchor? For an AI service, the core identity of sustainable operation is:

Gross margin    = (Revenue per request − Variable cost per request) ÷ Revenue per request
Unit economics  = Revenue per user − Variable cost per user

Unit price is set by market competition. What engineering teams can directly change is mainly variable cost and resource utilization.

3. AI Service Cost Structure: Where the Money Goes

You cannot optimize what you cannot see. AI service cost typically falls into five categories:

Cost category	Main components	Positively correlated with	Common metric
Model invocation	API fees; depreciation and power for self-hosted clusters	Total tokens, model tier	Cost per 1M tokens
Retrieval and data	Vector search, document parsing, indexing, storage	Knowledge base size, update frequency	Cost per 1K retrievals
Application-layer compute	Gateway, orchestration, tool calls, business logic	QPS, call-chain length	Requests per core-hour
Traffic and storage	Egress bandwidth, cross-region transfer, log and trace storage	Response size, retention period	GB-month cost
People and operations	Tuning, evaluation, on-call, security and compliance	System complexity, SLO requirements	Ops hours per 1K requests

Two points deserve emphasis.

First, tokens are not compute. Tokens are a billing and metering unit; compute is the physical resource. For the same one million tokens, the compute consumed can differ by an order of magnitude depending on context length, output volume, and whether a reasoning model is used.

Second, long context is a cost amplifier. For mainstream Transformer-style models, the complexity of attention computation generally grows quadratically with context length. Double the context length and the core computation grows roughly fourfold. At the same time, every decoding step has to read more of the historical cache, so read overhead grows approximately linearly. That is why pricing structures that charge more per output token have a sound basis in compute cost.

4. Optimization: Breaking Cost Down from Request to Resource

Cost optimization is not about finding one universally “cheapest” model. It is about finding the Pareto frontier between quality and cost. The work splits across six layers: request side, cache, model, inference engine, deployment resources, and application governance.

4.1 Request side: make the information leaner

The first step is to eliminate wasted computation. Common techniques:

• Context slimming: deduplicate and rerank retrieved results, truncate by relevance, compress conversation history, and strip redundant system prompts.
• Output constraints: specify the output format and a length cap to reduce useless generation.
• Fewer calls: merge tool invocations, shorten agent loops, and avoid repeatedly reading large files.

Sample code:

def build_prompt(query: str, retrieved: list[dict],
                 k: int = 5, max_chars: int = 3000) -> str:
    """Deduplicate + rerank + truncate: compress the context while
    keeping the key information."""
    seen, kept = set(), []
    for item in sorted(retrieved, key=lambda x: x["score"], reverse=True):
        text = item["text"].strip()
        key = text[:120]
        if key in seen:
            continue
        seen.add(key)
        kept.append(text)
    parts, budget = [], max_chars
    for text in kept[:k]:
        if budget <= 0:
            break
        parts.append(text[:budget])
        budget -= len(parts[-1])
    context = "\n".join(parts)
    return (
        "You are a rigorous assistant. Answer only from the material below; "
        "if the material is insufficient, say so explicitly.\n"
        f"Material:\n{context}\n\nQuestion: {query}"
    )

4.2 Cache layer: reuse computation you have already paid for

Caches fall into three categories by granularity:

Cache type	What is reused	Typical scenario	Main risk
Exact cache	Whole request and response	FAQs, fixed templates	Temperature must be 0; semantically similar but different questions cannot share an entry
Prefix cache	Internal state of the leading part of the prompt	Long system prompts, long-document QA, multi-turn sessions	Hit rate is extremely sensitive to wording
Semantic cache	Vector-similar question/answer pairs	Customer support QA	A false hit can return a wrong answer

Prefix caching deserves particular attention. The shared portion must be byte-for-byte identical, so dynamic content such as system prompts with timestamps and random IDs should go at the end. A prompt-template upgrade invalidates the cache, so roll it out gradually and monitor the hit rate. Context should be appended to, not rewritten. And cache effectiveness should be measured per model, per tenant, and per feature.

For semantic caching, set a similarity threshold, track a false-hit metric, run manual spot checks, and have a rollback plan. Calibrate the initial threshold by task type — for customer support QA, start testing somewhere between 0.85 and 0.92, then adjust based on observed error and hit rates.

Sample code:

def chat_messages(base_prompt: str, history: list[dict],
                  user_input: str) -> list[dict]:
    """Fixed shared prefix + append-only history: improves prefix cache hit rate."""
    return [
        {"role": "system", "content": base_prompt},
        *history,
        {"role": "user", "content": user_input},
    ]
def with_context(ctx_text: str, user_input: str) -> str:
    """Put dynamic content after the shared prefix so the cacheable part
    stays intact."""
    return (f"Answer from the material below; if it is insufficient, say so.\n"
            f"{user_input}\n\nMaterial:\n{ctx_text}")

4.3 Model layer: tiered routing and cascade calls

Different tasks place very different demands on model capability. Common tiering approaches:

• Route by request characteristics: assign a model tier based on task type, input length, and historical accuracy.
• Cascade calls: let a small model answer first, then decide based on confidence whether to escalate to a large model. The hard part is calibrating confidence; without calibration, hard cases get underestimated.
• Fast/slow thinking split: answer simple requests directly and trigger step-by-step reasoning only for complex ones. Reasoning models generate internal reasoning content that is usually billed as output tokens, so cost on long tasks can multiply — which makes on-demand triggering the right approach.

Sample code:

from dataclasses import dataclass
@dataclass
class ModelSpec:
    name: str
    price_in: float       # input price, CNY per 1M tokens
    price_out: float      # output price, CNY per 1M tokens
    max_input: int = 1_000_000
SPECS = {
    "small": ModelSpec("small-v1", price_in=1.0,  price_out=4.0),
    "large": ModelSpec("large-v1", price_in=10.0, price_out=30.0),
}
@dataclass
class Request:
    task: str
    input_tokens: int
    expected_output_tokens: int
def route(req: Request, conf: dict, stats: dict) -> ModelSpec:
    """Tiered routing based on task tier, context length, and the small
    model's historical accuracy.
    conf example:
    {
        "tier_by_task": {"summarize": "small", "code_review": "large"},
        "acc_floor": 0.90
    }
    stats example:
    {"summarize": 0.96, "code_review": 0.82}
    """
    tier = conf.get("tier_by_task", {}).get(req.task, "small")
    if tier == "small":
        if stats.get(req.task, 0.0) < conf.get("acc_floor", 0.90):
            tier = "large"
        elif req.input_tokens > SPECS["small"].max_input:
            tier = "large"
    return SPECS[tier]
def estimate_cost(req: Request, spec: ModelSpec) -> float:
    """Estimate the cost of a single call (CNY) from input/output tokens."""
    return (req.input_tokens * spec.price_in
            + req.expected_output_tokens * spec.price_out) / 1_000_000

The payoff from tiering depends on the shape of your traffic. Suppose that for a given task, the per-request cost of the small model is 34% of the large model’s, and a share p of traffic can be migrated. Total savings are then:

Total savings = Migratable traffic share × (1 − Small-model relative cost)

With a small-model relative cost of 34%:

• If 60% of traffic can be migrated, overall savings are about 39.6%.
• If 20% of traffic can be migrated, overall savings are about 13.2%.

Looked at only from the migrated requests, their cost drops by about 66%. Measuring the traffic breakdown first, then deciding which tier to invest in, is what keeps you from optimizing for optimization’s sake.

4.4 Inference engine layer: raise the effective output of each accelerator

Technique	How it works	Main benefit	Caveat
Continuous batching	New requests join the batch as soon as the previous step finishes	Throughput	Larger batches can raise queueing latency
Paged cache management	Allocate cache blocks on demand, reducing fragmentation	Memory utilization, concurrency	Clear gains with long context
Prefix caching	Reuse computation on shared prefixes	Input-side compute, time to first token	Requires exact matching and a fixed structure
Speculative decoding	A small model drafts, a large model verifies	Generation latency	Limited payoff when the acceptance rate is low
Quantization	Reduce the numerical precision of weights	Memory footprint, bandwidth pressure	Requires evaluating accuracy loss
Distillation	Train a small model on a large model’s outputs	Inference cost	The small model’s capability ceiling must be validated
Operator and compiler optimization	Operator fusion, efficient kernels	End-to-end throughput	Depends on engine support

Common utilization metrics:

GPU utilization (Busy) = Time the accelerator spends computing ÷ Total time
Memory bandwidth utilization (MBU) = Bytes actually read ÷ Theoretical peak bandwidth × 100%
Model FLOPs utilization (MFU) = Actual model FLOPs ÷ Theoretical peak FLOPs × 100%

The bottleneck in the generation phase is usually reading data, not computing. So high utilization does not necessarily mean high output. Queued requests keep an accelerator busy without improving throughput, and small-batch decoding can show very high utilization while producing less than a larger batch would. Capacity planning has to look at throughput, latency, and utilization together.

4.5 Deployment and resource layer: let resources follow the load

• Separate workload types. Online requests are latency-sensitive and batch jobs are not. They can share infrastructure, but they need priorities and quotas.
• Match resources to need. A stable baseline fits reserved or committed-use capacity; peaks fit elastic capacity; training and offline evaluation fit interruptible capacity.
• Scale ahead of demand. Expand based on historical patterns and recent trends, rather than waiting for an alert to fire.
• Shift load into the trough. Non-real-time work such as batch summarization, offline evaluation, and index rebuilding can run during low-traffic hours.
• Do not mix inference and training pools. Long training runs need continuous access to whole cards and high-speed interconnect; sharing a pool with online serving tends to hurt both.

4.6 Application and governance layer: make every dollar explainable

• Meter per request and per user: record tokens, cache hits, model tier, tenant, and feature for every call.
• Set quotas and budget alerts to stop a single user or a runaway loop from consuming large amounts of resources.
• Build a cost dashboard that tracks cost per 1M tokens, cost per user, resource utilization, and gross margin — displayed alongside quality metrics.
• Govern hidden tokens. Repeated tool calls by agents, duplicated log entries, and full trace retention all add up to costs that are easy to miss.

5. Measurement and Capacity Planning

5.1 Metrics system

Layer	Metrics	Purpose
Call layer	Input/output/reasoning tokens, cached tokens	Locate cost sources
Performance layer	Time to first token P95, inter-token interval P95, output tokens per second	Bound the optimization envelope
Resource layer	Accelerator utilization, memory usage, bandwidth utilization	Identify the bottleneck type
Quality layer	Task accuracy, escalation rate, human feedback	Prevent cost cuts from degrading quality
Economic layer	Cost per 1K calls, cost per active user, gross margin	Business decisions

Metric definitions must be consistent. “Cost per session,” for example, needs to state whether it includes multi-turn calls, and whether cached tokens are billed at the input price or the cache price must be labeled explicitly at the data source.

5.2 A rough cost model for self-hosted inference

The functions below estimate cost and capacity separately. They estimate accelerator cost from output tokens only; a real deployment still needs to factor in input tokens, memory, batching constraints, and load-test data.

import math
def estimate_unit_cost(gpu_cost_per_hour: float,
                       tokens_per_gpu_sec: float,
                       input_output_ratio: float = 3,
                       input_cost_weight: float = 1/3,
                       utilization: float = 0.6) -> float:
    """Estimate accelerator cost (CNY) per 1M output tokens."""
    effective = tokens_per_gpu_sec * utilization
    blended = effective / (1 + input_output_ratio * input_cost_weight)
    return gpu_cost_per_hour / blended * 1_000_000 / 3600
def estimate_capacity(daily_calls: int, out_tokens: int,
                      per_gpu_throughput: float, peak_ratio: float = 3,
                      target_util: float = 0.7) -> int:
    """Roughly estimate the number of accelerators needed from daily call
    volume. This estimate is based on output tokens only and does not fully
    cover input tokens, memory limits, or batching latency constraints."""
    peak_tps = daily_calls * out_tokens * peak_ratio / 86400
    return math.ceil(peak_tps / (per_gpu_throughput * target_util))

The all-in hourly cost of a single card usually includes hardware depreciation, power, data center space, network, and a share of operations. These parameters need real measurements accumulated over time; right after launch, load-test results can serve as initial values.

5.3 Queueing models: do not treat the average as the peak

The most common capacity-planning mistake is sizing resources to average load. Request arrivals are random. Under an M/M/1 approximation — service times roughly exponentially distributed, arrivals roughly Poisson, a single queue, and stable utilization — the following holds:

Average queue wait    ≈ Average service time × Utilization ÷ (1 − Utilization)
Average total response time ≈ Average service time ÷ (1 − Utilization)

“Average queue wait” here covers waiting only, not service time; use the average total response time when you care about user experience.

Utilization	Queue wait (relative)	Total response time (relative)
50%	1.0× service time	2.0× service time
80%	4.0× service time	5.0× service time
90%	9.0× service time	10.0× service time

In other words, when the accelerators “look busy,” the latency users experience may already be several times higher. Practical measures:

• Set the utilization alert threshold at 65%–75% instead of running near 90% for long stretches.
• Give long-output requests their own queue and timeout.
• Separate or tier batch jobs from online requests.

6. Closing the Validation Loop: How to Prove an Optimization Is Real

1. Establish a baseline: fix the evaluation set, sampling parameters, runtime environment, and model version.
2. Run A/B or shadow comparisons: run the new strategy in parallel with the old one and compare cost, quality, and latency.
3. Apply dual gates: if quality drops beyond an agreed threshold or a latency SLO is breached, the change does not ship.
4. Monitor continuously: model version updates, prompt changes, and shifts in traffic mix all invalidate historical conclusions.

Sample code:

def routing_stats(logs: list[dict]) -> dict:
    """Aggregate tiered-routing distribution, cost savings, and escalation
    rate from call logs."""
    total = len(logs)
    if total == 0:
        return {}
    by_tier = {}
    for item in logs:
        cur = by_tier.setdefault(item["tier"], {"count": 0, "cost": 0.0})
        cur["count"] += 1
        cur["cost"] += item["cost"]
    return {
        "total": total,
        "by_tier": by_tier,
        "escalation_rate": sum(x["escalated"] for x in logs) / total,
        "avg_cost": sum(x["cost"] for x in logs) / total,
    }

7. FAQ

7.1 What are AI service unit economics?

Unit economics looks at the difference between the revenue and the variable cost of a single call, a million tokens, or a single user. The point is not the total amount raised, but whether each call becomes more profitable as volume grows.

7.2 How do you calculate unit economics?

Use these definitions:

Gross margin per request = Revenue per request − Variable cost per request
Gross margin per user    = Revenue per user − Variable cost per user
Gross margin rate        = Gross margin per request ÷ Revenue per request

Variable cost per request should include model invocation, retrieval, storage, traffic, caching, and a necessary share of operations.

7.3 What drives unit cost up?

Long context, long outputs, reasoning models, multi-turn agent calls, low cache hit rates, low resource utilization, batch jobs crowding out online traffic, and full log retention can all raise cost.

7.4 How do you choose between caching and tiering?

Start with the shape of your traffic. High-frequency fixed Q&A suits an exact cache; long system prompts and multi-turn sessions suit prefix caching; semantically similar questions can use a semantic cache, but the false-hit rate must be controlled. Model tiering should be set based on task accuracy, input length, and cost differences.

7.5 What are the common mistakes?

Looking only at the per-token price; treating valuation multiples as proof of growth; overusing discount commitments; compressing context without evaluating the result; setting semantic-cache thresholds too loose; optimizing only the model and ignoring the call chain; and inverting the order of compression work.

8. Implementation Roadmap

Phase	Goal	Owner	Key deliverables	Acceptance metric
Month 1: See clearly	Establish a cost baseline	Architects, backend, data team	End-to-end call logs, token metering, cost dashboard	Core-path cost is attributable
Months 2–3: Cut cost	Ship low-risk optimizations	Backend, algorithms, platform	Context slimming, exact/prefix caching, quota alerts	Cost down, quality no lower than baseline
Months 4–6: Improve efficiency	Structural optimization	Algorithms, inference engine, SRE	Tiered model routing, cascade calls, engine tuning	Gross margin and SLOs improve together
Ongoing: Make it stick	Build a durable mechanism	Tech leads, finance, product	Cost SLOs, monthly reviews, benefit attribution	Cost is measurable, quality is verifiable

9. Conclusion

Valuations rise and fall on expectations, liquidity, and narrative — things engineering teams can rarely influence directly. What is actually within reach is the cost and quality behind every call, every accelerator, and every millisecond of latency.

When price becomes the new reference point, it is worth returning to three basic questions:

• What makes up the cost of this call?
• Which computations are being paid for more than once?
• Did quality hold while cost went down?

Answering those three questions clearly is more useful than predicting the next round’s valuation.