LLM Selection and Evaluation: A Practical Guide from Trillion-Parameter Models to Real-World Deployment
Trillion-parameter large language models are becoming the norm, but bigger parameter counts do not automatically translate into better business outcomes. This guide helps developers and technical decision-makers cut through launch-day hype and choose models based on reproducible evidence. It breaks down four key architectural mechanisms of this model generation—hybrid linear attention, attention residuals, sparse Mixture-of-Experts (MoE), and native vision understanding with long context—and explains their real trade-offs. It then lays out a three-layer evaluation system, methods for contamination control and statistical significance, an eight-dimension capability scoring matrix, and performance and cost testing across four load profiles. You will also find example evaluation and stress-test code, a layered cloud deployment architecture, a selection checklist, and five common pitfalls to avoid.
Introduction: When Trillion-Parameter Models Become the Norm
A new wave of trillion-parameter large models has arrived in quick succession. Terms like “sparsification,” “native multimodality,” and “hybrid linear attention” keep reshaping the technical narrative, while community discussions are full of phrases like “brute-force scaling” and “contending for the top tier.” One pattern is worth noting: model launches often coincide with important commercial milestones for the vendor, such as a new funding round or an IPO process. These milestones are perfectly legitimate, but they mean that external messaging tends to emphasize “capability leadership” and “industry standing.” That is exactly why the party doing the selection needs an independent, reproducible validation method.
For developers and technical decision-makers, the real question is not “whose model has more parameters,” but rather:
- Can the new architectural mechanisms reliably deliver value in your specific business scenarios?
- Does the capability gain justify the costs of migration, inference, and operations?
- Can the evaluation conclusions be reproduced, rather than resting on a few polished demos?
A demo shows a “hand-picked best-case sample.” An evaluation requires a complete evidence chain that is repeatable, comparable, and regression-testable. This article does not revolve around any single product. Instead, it distills the common characteristics of the trillion-parameter generation and builds a methodology that takes you from “understanding the architecture” to “selecting with data” to “deploying reliably.”
Part 1: Understanding Next-Generation LLMs Across Four Key Technical Dimensions
1.1 Hybrid Linear Attention: Balancing Long-Sequence Efficiency and Global Modeling
The computational cost of standard self-attention grows quadratically with sequence length, so training and inference costs rise sharply in long-context scenarios. Linear attention reduces the computational complexity to roughly linear in sequence length. Its core idea is to replace the ever-growing key-value cache with a fixed-size “state matrix,” which substantially lowers memory usage and decoding overhead. But its weakness is equally clear: a fixed-capacity state cannot fully preserve all historical information, so on tasks that require precise retrieval of distant details, it tends to “lose information.”
Newer hybrid linear attention approaches in the industry typically improve on this from two directions:
- State update rules: Borrowing the idea of incremental updates from the “delta rule,” the model selectively corrects or forgets existing state based on the current input instead of simply accumulating it, strengthening its memory of critical information.
- Hybrid layer design: Using efficient linear attention in most layers while retaining full attention in a few layers, trading a controlled amount of computation to recover precise global retrieval capability.
The value of this kind of architecture in long-context scenarios shows up in two ways:
- During autoregressive decoding, there is no need to cache the full set of key-value pairs, which significantly reduces peak memory usage and per-step decoding computation.
- Under mixed workloads of long and short inputs, the throughput curve degrades more gracefully.
But “linear” does not mean “free.” Reading and writing the state matrix introduces extra memory-access overhead, and the architectural advantage is not uniform across different sequence lengths. So an evaluation cannot rely on theoretical complexity alone; it must run real measurements within your target length range.
Selection focus points:
- Hardware compatibility: Can the custom attention operators run efficiently on mainstream accelerators, and is there a fallback path to a generic implementation?
- Hybrid ratio: The proportion of linear layers to full-attention layers, and where the full-attention layers are positioned.
- Capability validation: Use real business “needle-in-a-haystack” and multi-hop retrieval tasks as the yardstick, paying particular attention to performance when the key information sits in the middle of a long text.
1.2 Attention Residuals: Information Pathways in Deeply Stacked Networks
The attention mechanism is the core component of Transformer-style architectures, responsible for information exchange between tokens. Once you stack dozens or even a hundred layers, each layer’s output together with its residual connection forms a vertical pathway for information. As the number of layers grows, gradients tend to vanish or explode during backpropagation, and the feature representations of different layers tend to become homogenized.
The idea behind Attention Residuals is to preserve explicit cross-layer information pathways deeper in the network, so that details extracted by early layers can influence the final output more directly rather than being diluted as they propagate layer by layer. You can think of it as a further reinforcement of residual connections for the era of deeper networks.
Evaluation points:
- On tasks requiring multi-step reasoning (math, logic, complex instructions), does it show a more stable intermediate reasoning chain?
- Are the gains from added depth offset by training-stability problems? That is, does the model converge easily, or is there layer-to-layer “collapse”?
- For downstream use, the practical significance is whether the inference stage can reduce long outputs that repeatedly “go in circles,” thereby lowering output token consumption.
1.3 Sparse Expert Architecture: Decoupling Parameter Scale from Inference Cost
Parameter count is often treated as an intuitive yardstick for model capability, but for actual inference the more critical factor is “how many parameters must be activated in a single forward pass.” Today’s trillion-level models generally adopt a Mixture-of-Experts (MoE) architecture: the total parameters provide knowledge capacity, while the sparse activation mechanism determines how many parameters are actually used per computation. Take a model that activates roughly 3% of parameters, using hundreds of billions of activated parameters to support a trillion total, as an example. Here the model’s knowledge capacity and inference cost are decoupled, making it possible for the first time for “big” and “cheap” to hold at the same time.
The trade-offs come along with it:
- Memory pressure: During inference, all expert parameters must remain resident in memory, so memory capacity and bandwidth become the new bottleneck.
- Expert-parallel communication: Large-scale deployment requires expert parallelism, which significantly increases inter-node communication overhead. Whether the routing load is balanced directly determines actual throughput.
- Fine-tuning difficulty: Full fine-tuning is costly, so parameter-efficient methods such as low-rank adaptation are usually relied upon, and adding new experts can easily trigger training instability.
- Performance variability: Differences in computation patterns caused by routing can produce a long-tail latency distribution across different requests.
Therefore, “trillion parameters” is not a conclusion in itself; it only indicates that the model has a large upper bound on knowledge capacity. Actual capability depends on training data quality and scale, post-training strategy (preference optimization, long chain-of-thought reinforcement, tool-use training, and so on), and inference engineering, all of which must be verified through real measurement.
1.4 Native Vision Understanding and Long Context: The Engineering Implications of Unified Input
The mainstream multimodal approach in the past was a “visual encoder + projection layer + language model” concatenation, where visual information was projected and then fed into the language model, and the two were not fully aligned in representation space or training objectives. Native vision understanding instead has text and image data jointly participate in training from the pre-training stage, so visual and language signals are aligned within the same representation space.
The direct value to developers shows up in the following ways:
- More natural multimodal instruction following: You can ask continuous follow-up questions about the same image and alternate between text and image conditions, just as in a pure-text conversation.
- More reliable fine-grained perception: Performance is more stable on tasks such as document layout analysis, chart reading, and UI screenshot localization.
- More unified invocation: There is no need to switch between text and vision interfaces, which reduces adaptation costs at the application layer.
Long-context capability likewise needs to be reconsidered. “Supporting a given context length” is a specification metric; “still being able to accurately locate and use information at that length” is a capability metric. In practice, the phenomenon where the model misses key information located in the middle of a long text (the so-called “lost in the middle”) remains fairly common. When native vision understanding is combined with long context, the typical scenario is “dozens of pages of scanned or technical documents plus multi-turn follow-up questions,” which raises the bar for evaluation methods: they must simultaneously examine cross-position retrieval, image-text cross-reasoning, and multi-turn memory retention.
Part 2: Building a Reusable Evaluation System So Selection Is Evidence-Based
2.1 A Three-Layer Evaluation System
Conclusions from any single source are unreliable. It is recommended to build a three-layer, progressive evaluation system:
| Layer | Evaluation target | Primary role | Update cadence |
|---|---|---|---|
| L1 Public benchmarks | Public evaluation sets for general knowledge, reasoning, code, etc. | Cross-comparable, fast screening | Updated with model releases |
| L2 Domain evaluation set | A private set sampled from real business requests and manually labeled | Measures business fit | Expanded as the business iterates |
| L3 End-to-end A/B | Online experiments on real traffic | Validates real-world gains | Continuous |
On L1: The strength of public leaderboards is cross-comparability; the weaknesses are data contamination and question saturation, and they are often disconnected from business scenarios. It is best to treat public benchmarks as an “admission threshold” rather than a “final basis.” For example, a model that leads on a general code benchmark is not necessarily equally strong under your internal code frameworks and coding standards.
On L2: The domain evaluation set is the core basis for selection. When building it, follow these principles:
- Real sources: Sample real requests from production logs, covering both typical distributions and long-tail scenarios.
- Clear labeling: Define a gold answer or scoring rubric for each question, and remove or split questions with ambiguous boundaries.
- Difficulty tiering: Set easy, medium, and hard tiers to avoid concentrating all questions in the same difficulty range, which would make it hard to “tell models apart.”
- Continuous expansion: Regularly fold new problem types observed online into the set to prevent the evaluation set from “aging.”
2.2 Contamination Control and Statistical Significance
- Contamination control: Use real business data generated after the model’s training cutoff; paraphrase public questions or add custom business constraints; embed unique random markers in prompts to detect training memorization.
- Sampling configuration: Fix decoding parameters (such as temperature, top_p, maximum output length, and random seed) to ensure a fair comparison.
- Repeated sampling: For capability tasks, repeat sampling 3–5 times per question, and report both first-attempt pass rate and any-pass rate.
- Distinguish capability variance from random variance: The former can be mitigated through self-consistency voting, majority voting, or majority-path fusion; the latter requires a larger sample size to eliminate.
- Paired comparison: Use paired statistical methods (such as computing confidence intervals with a paired bootstrap, and using the McNemar test to judge significance) rather than simply comparing average scores. If the score difference between two models falls within the confidence interval, you should not draw a conclusion based on it.
2.3 Prompt Consistency, Judging, and an Automated Pipeline
Inconsistent prompt formatting is a common source of evaluation bias. Different models prefer different chat templates, system prompts, and function-calling formats, so using a single generic prompt to evaluate all models is not entirely fair. It is best to tune each candidate model under its official recommended format separately and record the final version used as the baseline for subsequent regression comparisons.
The scoring method can be chosen by task type:
- Objective tasks (extraction, classification, computation): Judge automatically with rules.
- Open-ended tasks (writing, solution generation): Use “model judge + manual spot-checks,” and clearly specify the scoring dimensions to the judge model.
- Model judging: Control for position bias and length preference; you can randomly swap the order of candidates and debias the results.
- High-risk tasks (medical, financial, legal advice): Domain expert review must be introduced.
The evaluation sets and evaluation code are best managed as structured configuration:
# eval_suite.yaml: Example evaluation suite configurationsuite_name: "enterprise_qa_v3"sampling:temperature: 0.0 # Capability evaluation prioritizes reproducibilitymax_tokens: 2048repeats: 3 # Number of repeated samples per questiontasks:- name: knowledge_qafile: datasets/qa.jsonlscorer: exact_matchweight: 0.2- name: document_extractionfile: datasets/extract.jsonlscorer: field_f1 # Compute F1 per fieldweight: 0.3- name: code_generationfile: datasets/code.jsonlscorer: unit_test # Run unit tests in an isolated sandboxweight: 0.3- name: long_contextfile: datasets/longctx.jsonlscorer: llm_judge # Model judge; requires manual spot-checksweight: 0.2output:report_dir: reports/compare_with: baseline_v2 # Paired comparison against the baseline version
Illustrative code:
import hashlibimport jsonimport statisticsimport urllib.requestfrom concurrent.futures import ThreadPoolExecutor, as_completedENDPOINT = "https://your-gateway.example/v1/chat/completions" # Replace with the actual evaluation gateway addressdef ask(question: str, config: dict) -> dict:payload = {"model": config["model"],"messages": [{"role": "user", "content": question}],"temperature": 0.0,"max_tokens": config["max_tokens"],}req = urllib.request.Request(ENDPOINT,data=json.dumps(payload).encode("utf-8"),headers={"Content-Type": "application/json","Authorization": "Bearer <token>"}, # Inject the key via environment variables; never hardcode it)with urllib.request.urlopen(req, timeout=120) as resp:data = json.loads(resp.read().decode("utf-8"))usage = data.get("usage", {})return {"answer": data["choices"][0]["message"]["content"],"prompt_tokens": usage.get("prompt_tokens"),"completion_tokens": usage.get("completion_tokens"),}def run_task(task_file: str, config: dict, scorer) -> dict:with open(task_file, encoding="utf-8") as f:cases = [json.loads(line) for line in f if line.strip()]def run_case(case):# Append a unique marker to the prompt to detect whether the model has memorized the evaluation dataquestion = f'{case["question"]}\n[request_id:{hashlib.md5(case["id"].encode()).hexdigest()}]'results = [ask(question, config) for _ in range(config["repeats"])]scores = [scorer(r["answer"], case["gold"]) for r in results]return {"id": case["id"],"scores": scores,"first_pass": scores[0],"pass_any": max(scores),"tokens": [r["completion_tokens"] for r in results if r["completion_tokens"]],}with ThreadPoolExecutor(max_workers=config["concurrency"]) as pool:futures = [pool.submit(run_case, c) for c in cases]records = [f.result() for f in as_completed(futures)]n = len(records)return {"task": task_file,"n": n,"first_pass_rate": statistics.mean(r["first_pass"] for r in records),"pass_any_rate": statistics.mean(r["pass_any"] for r in records),"avg_output_tokens": statistics.mean(statistics.mean(r["tokens"]) if r["tokens"] else 0 for r in records),"records": records, # Keep per-question details for paired significance testing}
After the run finishes, aggregate each task’s score into a composite score by weight, while keeping per-question details for subsequent paired significance testing.
2.4 An Eight-Dimension Capability Scoring Matrix
| Capability dimension | Typical test method | Key metrics to observe |
|---|---|---|
| Logical reasoning & math | Multi-step word problems, logic puzzles, constraint problems | First-attempt pass rate, step verifiability, self-consistency |
| Code & tool use | Enterprise-codebase generation, unit tests, function-call parsing | Test pass rate, compile rate, parameter-extraction accuracy, invalid-call rate |
| Long context | Long-document summarization, cross-position retrieval, multi-document comparison | Recall at different positions, answer faithfulness, hallucination rate |
| Vision understanding | Document layout analysis, chart reading, screenshot localization | Field-extraction accuracy, digit-reading error, image-text consistency |
| Instruction following & output structure | Fixed JSON/table output, multi-constraint combined instructions | Format compliance rate, constraint-satisfaction rate, retries needed per generation |
| Chinese & multilingual | Chinese long-form writing, terminology translation, dialect and colloquial understanding | Human ratings, terminology consistency, fluency |
| Hallucination & factuality | Trap-laden Q&A, enterprise knowledge Q&A | Grounded-answer rate, reasonableness of refusals, citation traceability |
| Safety & compliance | Sensitive content, jailbreak inducement, prompt injection | Block rate, false-positive rate, unauthorized tool-call block rate |
The composite score should not be a simple sum. It is best to weight it according to business priorities and to set “veto” items. For example, if the function-call parsing accuracy on a code-generation task is too low, the model should not be the top choice no matter how high it scores on other dimensions.
Part 3: Performance and Cost Evaluation: Don’t Be Misled by “Average Latency”
3.1 First, Clarify Three Concepts
- Time to first token (TTFT): The time from sending a request to receiving the first output token. It is mainly determined by input length (prefill computation) and queuing, and it directly affects the conversational experience.
- Time per output token (TPOT): The average time to generate each output token in the decoding stage, determined by decoding computation and memory bandwidth. It affects the perceived “reading speed.”
- Throughput: The total number of tokens or concurrent requests processed per unit of time, which determines cluster size and cost.
Most people focus only on average latency during evaluation, but user experience depends on the latency distribution. Under concurrency pressure, degradation in tail latency (such as the 95th and 99th percentiles) is often more damaging than a rise in the average.
3.2 Stress Testing Across Four Load Combinations
| Load type | Input / output | Representative scenario | Core bottleneck |
|---|---|---|---|
| Long input, short output | 8K–128K / <512 | Document summarization, contract review | Prefill computation, KV cache |
| Short input, long output | <1K / 4K–32K | Code generation, long-form writing | Decoding throughput, memory bandwidth |
| Mixed long and short | Mixed distribution | General business traffic | Scheduling strategy, batching efficiency |
| High-concurrency bursts | Any | Marketing campaigns, scheduled jobs | Rate limiting, queuing, elasticity of scaling |
When stress testing, distinguish between two scenarios: public API calls measure the provider’s rate limiting and quotas; private deployment must be assessed together with the hardware configuration. Performance differences between same-tier accelerators from different vendors are significant, so you cannot directly reuse someone else’s stress-test data.
Illustrative code:
import timeimport urllib.requestimport jsonfrom concurrent.futures import ThreadPoolExecutor# Endpoint and key are injected via environment variablesENDPOINT = "https://your-gateway.example/v1/chat/completions"def measure_latency(filler: str, prompt: str, max_tokens: int) -> dict:"""filler builds an input of a specified length; prompt is the real business question"""payload = json.dumps({"messages": [{"role": "user", "content": filler + prompt}],"max_tokens": max_tokens,"stream": True, # Streaming output is required to measure TTFT accurately}).encode("utf-8")req = urllib.request.Request(ENDPOINT, data=payload,headers={"Content-Type": "application/json"},)start = time.perf_counter()ttft, output_tokens = None, 0with urllib.request.urlopen(req, timeout=300) as resp:for raw in resp:line = raw.decode("utf-8").strip()if not line.startswith("data:"):continuechunk = line[5:].strip()if chunk == "[DONE]":breaktry:delta = json.loads(chunk)["choices"][0].get("delta", {})except (json.JSONDecodeError, KeyError, IndexError):continueif delta.get("content"):if ttft is None:ttft = time.perf_counter() - start # Timestamp when the first content chunk arrivesoutput_tokens += 1 # Illustrative: in practice, count full tokenstotal = time.perf_counter() - startreturn {"ttft_s": ttft,"tpot_ms": (total - ttft) * 1000 / max(output_tokens - 1, 1),"total_s": total,"output_tokens": output_tokens,}def stress_test(concurrency: int, filler_len: int, prompt: str, max_tokens: int):with ThreadPoolExecutor(max_workers=concurrency) as pool:futures = [pool.submit(measure_latency, "filler text" * filler_len,prompt, max_tokens) for _ in range(concurrency * 3)]results = [f.result() for f in futures]ttfts = sorted(r["ttft_s"] for r in results if r["ttft_s"])return {"concurrency": concurrency,"input_chars": filler_len,"ttft_p95_s": ttfts[int(len(ttfts) * 0.95)] if ttfts else None,"ttft_p99_s": ttfts[int(len(ttfts) * 0.99)] if ttfts else None,"throughput_tokens_per_s": sum(r["output_tokens"] for r in results)/ sum(r["total_s"] for r in results),}
3.3 Cost Modeling and Optimization
Plugging stress-test data into the following cost model makes it easier to compare candidate models directly:
Cost per request = input tokens × input price × (1 - prefix cache hit rate)+ output tokens × output priceMonthly total cost ≈ cost per request × monthly request volume × (1 + retry rate)+ long-context surcharge+ manual review cost
A model’s output style also affects cost. A “verbose-leaning” model may consume more output tokens on the same task, and output tokens are usually priced higher than input tokens. So cost evaluation should be based directly on measured average input/output token counts, not theoretical estimates.
Common optimization techniques include:
- Prefix caching: Enable caching for fixed prefixes such as system prompts and knowledge-base templates to reduce the cost of repeated inputs and lower TTFT.
- Batching and asynchronous tasks: For offline document processing and batch summarization, use asynchronous batch APIs to trade throughput for unit price.
- Structured output constraints: Use constrained decoding to keep JSON format stable and reduce parsing failures and retries.
- Inference acceleration such as speculative decoding: Improve decoding throughput in private-deployment scenarios.
- Model tier routing: Route simple tasks to lightweight models, and call the flagship model only for complex tasks.
- Quantized deployment: Reduce memory usage and deployment cost while keeping accuracy loss acceptable.
Part 4: Deployment Architecture: A Layered Design for General Cloud Capabilities
The conclusions from capability evaluation must land in a system that runs stably. It is recommended to build it in the following layers, and to prioritize mature general cloud capabilities (such as model serving, object storage, message queues, container platforms, logging services, monitoring and alerting, and so on) to reduce the cost of building your own infrastructure.
- Access layer: A unified model gateway handles authentication, rate limiting, quota management, multi-model routing, and degradation and circuit breaking. When the primary model times out or its error rate rises, the gateway automatically switches to a backup model or returns a cached result.
- Inference layer: Managed model serving suits rapid validation and businesses with pronounced elasticity; private inference suits data-sensitive scenarios or stable-volume workloads that pursue optimal unit cost. You can deploy inference services on a container platform and combine elastic compute with batch jobs to handle offline workloads.
- Data and knowledge layer: Use object storage to hold raw documents and evaluation datasets; use message queues to smooth out spikes in asynchronous tasks (batch summarization, document parsing); and combine general-purpose vector retrieval to implement retrieval-augmented generation, feeding precise knowledge snippets to long-context models.
- Application layer: Responsible for prompt template management, structured output parsing, function-call orchestration, and result caching.
- Observability and regression layer: Use logging services to collect request latency, token consumption, error codes, and model outputs; use monitoring and alerting to set thresholds for latency and error rate; and continuously expand the evaluation set from logs to form a closed loop of “evaluate — deploy — feedback — re-evaluate.”
Below is an example of multi-model routing:
class ModelRouter:"""Select a model by task complexity and automatically degrade on abnormal requests"""def __init__(self, primary, fallback, light):self.primary, self.fallback, self.light = primary, fallback, lightself.fail_count = 0def route(self, task: dict) -> str:if task.get("type") in {"chat", "classification"} and task.get("tokens", 0) < 1024:return self.light.name # Lightweight tasks use the light modelreturn self.primary.name # Complex tasks use the flagship modeldef call(self, task: dict):model = self.primary if self.route(task) != self.light.name else self.lighttry:result = model.invoke(task)self.fail_count = 0return resultexcept (TimeoutError, ConnectionError):self.fail_count += 1# Trigger fallback and alerting once consecutive failures reach the thresholdif self.fail_count >= 3:return self.fallback.invoke(task)raise
Part 5: A Selection Checklist and Common Pitfalls
5.1 Selection Checklist
- Define business constraints: Data compliance, latency budget, cost ceiling, and whether private deployment is required.
- Narrow the candidate set: Using public evaluations and community feedback, shortlist 3–5 candidate models.
- Build a domain evaluation set: Based on real business logs, construct a private set of at least several hundred questions.
- Run capability evaluation: Cover all eight dimensions and produce a composite score along with any veto-item conclusions.
- Stress test and estimate cost: Measure latency distribution and throughput across the four load types, and plug the results into the cost model.
- Run a small-scale canary: Choose 5%–10% of traffic for an A/B experiment and monitor business metrics.
- Establish a regression mechanism: Fold evaluation into the CI pipeline so that every model upgrade is automatically regression-tested beforehand.
5.2 Five Common Pitfalls
- Leaderboard-only thinking: A high public score does not equal strong business performance; leaderboards suffer from contamination and saturation.
- The demo effect: Demo cases are carefully selected and cannot reflect performance on long-tail scenarios.
- Parameter-only thinking: A trillion parameters is only a capacity ceiling; final capability is jointly determined by data, post-training, and engineering.
- Looking only at average latency: Ignoring tail latency and long-input scenarios will make the production experience noticeably worse than the stress-test expectation.
- Overlooking migration costs: Hidden costs such as rewriting prompts, adapting function calls, and reworking output parsing can sometimes exceed the cost difference between the models themselves.
Conclusion: Turning “Glory” into Measurable Engineering Value
The emergence of trillion-parameter models marks a shift in the LLM competition from “parameter scale” to a combined contest of “mechanism innovation + engineering efficiency + scenario deployment.” Mechanisms such as hybrid linear attention, attention residuals, sparse expert architecture, and native vision understanding have opened new space for long context, multimodality, and low-cost inference. But the value of these mechanisms must ultimately be tested by real business.
For developers, building a reproducible evaluation system delivers more long-term value than chasing any single-point conclusion:
- Use layered evaluation to distinguish “can demo” from “can go live.”
- Use statistical methods to avoid being misled by random variance.
- Use stress testing and cost models to calculate the input-output trade-off clearly.
- Use general cloud capabilities to carry elasticity, stability, and observability.
When every model upgrade can be quantified, regression-tested, and tracked, the “glory” of technical progress can truly be converted into sustainable business value.
