Training-Time Capability Limits and Tiered Access for Large Language Models: Can Evaluation Results Be Trusted?

1. The Short Answer

An evaluation report answers a narrow question: how a model scored, or how many tests it passed, under a specific question set, inference budget, and prompt template. It does not prove that a model is safe in every scenario, and it does not replace an enterprise’s own access audit. When reading an evaluation report, first check which scenarios it covers, then decide which conclusions it can support.

This article addresses three questions: how training-time methods can actively suppress some capabilities, how one base model can be released in tiers based on permissions, and how to interpret evaluation results when a model may recognize that it is being tested. The end of the article provides a risk-tier table, a checklist of evaluation criteria, and statistical steps for measuring over-blocking and under-blocking.

2. Training-Time Restrictions: What They Can and Cannot Do

Training-time constraints reduce a model’s output capability in certain directions during training or post-training. They use methods such as data filtering, preference optimization, and reward design in reinforcement learning. Three common approaches:

• Data side: reduce the weight of detailed technical content in certain categories in the training corpus, or remove it.
• Objective side: mark specific types of responses as unacceptable in the reward model or preference data.
• Evaluation side: test repeatedly with dedicated red-team question sets, and retrain to suppress the capability again whenever it reappears.

Training-time constraints differ from inference-time filtering. Inference-time filtering is an external component that can be replaced and upgraded separately. Training-time constraints are encoded in the model weights and usually require retraining or fine-tuning to change, so their reliability depends more on the quality of training records. The two approaches can be combined. The external layer can mitigate some risks that training-time constraints do not cover.

This approach has three limits.

1. Suppression is probabilistic. The model may still output suppressed content on adjacent topics, when a question is phrased differently, or in multi-turn follow-ups. Passing a single test does not show that the capability does not exist. Passing a single test also does not show that the capability has been eliminated.
2. Suppression affects legitimate capabilities. After excessive suppression, the model may refuse or give vague answers in valid settings such as medicine, security research, and education. This increases over-blocking.
3. Suppression cannot replace access control. Organizations with legitimate needs may find that a suppressed model cannot support their work. An access management system must provide a separate channel for them.

To address over-blocking, teams should do three things. First, set up an appeal process so legitimate users can explain their purpose and submit supporting documents. Second, for verified organizations, set scenario allowlists at the access layer. Identity and contract should determine the scope of access, instead of loosening the suppression in the model itself. Third, log every refusal, appeal, and outcome to form an audit trail. These records also serve as the basis for assigning responsibility and tracing versions.

Records that engineering teams should keep include the target categories of suppression, the versions of the data and question sets used, capability comparisons before and after suppression, and regression test results after each retraining.

3. Tiered Access: One Base Model, Opened by Permission

One set of base weights can produce several release forms. The base capabilities are the same across forms. The main differences are the safety filtering layer at inference time and the access rules. A typical structure has three layers:

Layer	Responsibility	Typical failure mode	Observable metrics
Base capability layer	General reasoning, code, document understanding	Capability regression, inconsistent performance across release forms	Version-to-version differences in benchmark scores
Safety filtering layer	Classify requests before and after inference, identify high-risk requests	Under-blocking, over-blocking, classifier bypass	Block rate, over-blocking rate, under-blocking rate, confidence distribution
Access control layer	Decide whether to allow requests based on identity, purpose, and contract	Expired credentials, misconfigured permissions, unauthorized access	Approval rate, number of permission changes, number of unauthorized access attempts

The benefit of this structure is reuse. The base model is developed once, and risk controls are iterated in the outer components. The cost is higher system complexity, and an error in any layer can affect the others. For example, when the over-blocking rate of the safety filter rises, the access control layer receives many appeals, and manual review workload increases. When the base capability layer changes version, the filter thresholds may need to be recalibrated.

Each layer has a defined exception procedure.

• Expired credentials: automatically freeze the corresponding permissions, notify the organization to renew, and restore access after the renewal documents pass review.
• Unauthorized access attempts: record the source and time of the request, trigger an alert, and have the security lead review it.
• Misconfigured permissions: roll back to the previous configuration, record who made the change and why, and update the configuration review process after a postmortem.

The access decision can follow these steps:

1. Verify the applicant’s identity and the organization’s credentials.
2. Confirm the purpose and contract scope, and determine whether the request falls under a restricted scenario.
3. Match an access channel that can be opened for the risk level.
4. Set rate limits and log retention rules.
5. Record approval materials and validity period, and send a reminder before expiry to schedule review.
6. If review fails or renewal is not completed by expiry, revoke access and keep the records.

Each step needs an assigned responsible role, usually the access owner, the security lead, and the business lead. Unclear responsibility at any step is often the hardest part to trace when access goes wrong.

4. Evaluation Awareness: Interpreting Results When a Model Detects a Test

A model may be able to recognize that it is in an evaluation. If the model behaves more cautiously when it detects a test context, the test results will overestimate its performance in real use. Conversely, if the model behaves abnormally during testing, the question set may differ substantially from real scenarios. Either way, evaluation conclusions need a statement of their scope.

To judge whether an evaluation report is credible, check the following:

• Question set source: whether it was built internally, comes from an external public source, or was supplied by the developer. Public question sets may already be in training data, so an overlap check is needed.
• Prompt template: whether it resembles real usage, whether it includes a system prompt, and whether the template has a version number.
• Test method: whether it used single-turn or multi-turn conversations, whether it includes re-tests with reworded questions, and whether it includes samples from real-world scenarios.
• Independence: whether the evaluator and the developer are the same entity. Self-assessed results should be spot-checked by a third party.

Cross-validation methods include expressing the same type of question in several wordings, testing consistency across multi-turn conversations, and, with authorization and privacy protections in place, sampling production requests after redaction. Passing once does not mean the model is safe. It only shows that the model met expectations under the current question set and conditions.

5. Risk Tiers and Evaluation Reporting Checklist

The table below assigns access levels by scenario risk. Each level lists the main over-blocking and under-blocking risks, access requirements, and fallback actions, so the table can be mapped directly to permission settings.

Level	Typical scenarios	Over-blocking risk	Under-blocking risk	Access requirements	Fallback action
R1 General, low risk	Document organization, general code, everyday Q&A	Low	Low	Verified account, standard user agreement	Standard rate limiting, anomaly logging
R2 Specialized, restricted	Medicine, law, security research, education	High	Medium	Organization credentials, stated purpose, contract terms	Switch to manual review channel, suspend automatic approval
R3 High risk	Technical content that could cause serious harm	Medium	High	Dedicated approval, two-person review, independent audit	Suspend the channel, freeze related permissions, start a postmortem

Before citing an evaluation result, log each item below:

• Question set: name, version number, source, whether it is public, and results of the overlap check with training data.
• Inference budget: maximum output length, sampling parameters, and whether tool calls and multi-turn conversations are allowed.
• Prompt template: template version, system prompt content, and whether it matches the production environment.
• Scoring crite