Open-Weight Model Selection: Key Evaluation Criteria from Parameter Scale to Commercial Licensing

Open Weights Don’t Mean Open Commercial Use

The “open-weight” label is increasingly entering the shortlisting process for teams evaluating large models. A very large model becomes attractive to teams with reasonable infrastructure capability when three conditions hold at once: the weights are downloadable, the license boundaries are clear, and inference costs can be estimated. In practice, these three conditions are rarely all verified carefully.

The model discussed here serves as a typical reference case: roughly 2.8 trillion total parameters with a sparse activation architecture, about 104 billion parameters activated per token, and a context window of around 1 million tokens. The weights ship as MXFP4-precision shards totaling roughly 1.56TB. On paper, it continues the design direction of the previous generation in the same family, with no aggressive expansion in parameter count. The real change is in distribution: the weights are no longer confined to a technical report but released in an accessible form.

There is, however, a clear distance between “downloadable” and “commercially usable.” License restrictions need to be unpacked clause by clause:

• If you offer the model as a hosted service and your revenue exceeds $20 million over 12 consecutive months, a separate agreement is required.
• If monthly active users exceed 100 million or monthly revenue exceeds $20 million, the model name must be clearly displayed in the product interface.
• Internal use and calls through the official API are not subject to these clauses.

For small and mid-sized teams, internal deployments face little friction—but compliance costs climb sharply once business scale crosses those thresholds.

What Benchmark Scores Actually Tell You

On the capability side, the officially reported Terminal-Bench 4.0 score is 65.7%—slightly below a leading closed-source model at 66.4% and above another frontier model at 58.2%. A gap of less than one percentage point sounds negligible.

Several layers need to be separated here. First, this is a vendor self-reported result obtained with inference intensity pushed to a high level, not an independent reproduction by a third party under identical evaluation conditions. Second, a single benchmark reflects only one capability dimension: Terminal-Bench focuses on terminal operation and task execution, and its results cannot be extrapolated to code generation, long-document understanding, or mathematical reasoning. Third, the gap between leaderboard scores and real-world project performance is often far larger than the leaderboard gap between models.

The safer reading is this: the model has entered the top tier on this particular benchmark, but whether it “catches up” with a given closed-source product in real business scenarios must be re-validated on your own evaluation set. Selection decisions should rest on backtest data from your own workloads, not on vague claims like “close to model X.”

The Structural Cost Gap Between API Calls and Self-Hosting

Open weights bring one direct benefit—the possibility of private deployment—but that doesn’t mean the cost is necessarily lower than calling an API.

Using the official API price as a reference, output costs about $10 per million tokens. On the same day, another vendor released a comparable model with output pricing around $2.5 per million tokens—roughly a fourfold difference. The latter has about 770 billion total parameters and 49 billion activated parameters, and its full weights have not been publicly released. In the short term, third-party APIs remain the better option for cost-sensitive developers.

Cost accounting for private deployment is far more complex. Hosting 1.56TB of weights requires, at minimum: enough VRAM or a VRAM-plus-RAM hybrid setup, high-speed storage to load the sharded weights, mature MXFP4 quantization support in the inference framework, and ongoing operational investment to keep the service running. If a team’s call volume is insufficient to amortize hardware depreciation and labor costs, private deployment does not make financial sense.

A simplified way to compare:

Private deployment monthly cost = hardware depreciation/month + ops labor/month + datacenter or cloud resources/month
API monthly cost = monthly token volume × price per million tokens

Only when the monthly cost of private deployment is significantly lower than API costs—and the team can continuously maintain the inference service—does self-hosting make practical sense. Otherwise, open weights provide “optionality,” not a “cost advantage.”

Safety Disclosures Deserve More Attention Than Leaderboards

One type of information is more valuable than performance data during selection: safety disclosure. The official technical notes state explicitly that after cyberattack-related training data was removed, the pass rate on exploit tests dropped from 90.5% to 81%. That number carries two key signals.

First, the model does possess some exploit knowledge—a risk that security teams must assess before adoption. For an enterprise using the model in security testing or red-team scenarios, this is a capability; for a conversational product serving ordinary users, it becomes a misuse risk.

Second, training data composition directly affects performance on specific tasks. A drop of nearly 10 percentage points after removing one data category shows that the model’s exploit capability is highly correlated with its training data rather than pure reasoning generalization.

Before integrating or deploying the model, two assessments are recommended at minimum: permission boundary testing against your own business scenarios, to confirm whether the model can easily be induced into sensitive operations; and a clear definition of which outputs require additional filtering or human review. Safety testing is not “running a public attack sample set once”—it should simulate real user input patterns along your actual product interaction paths.

Recommendations and an Adoption Path

Overall, evaluating an open-weight model shouldn’t start with “is it open,” but with four independent dimensions:

1. Capability match: performance on your own evaluation set, not a single public leaderboard.
2. License boundaries: whether current business scale and growth expectations over the next 12 months could trigger restriction clauses.
3. Cost structure: a long-term comparison of total private deployment cost versus API costs.
4. Security and compliance: model behavior on sensitive tasks, plus whether internal monitoring and audit capabilities are sufficient.

For most developers, the safer path is to validate model capabilities through the API first, then evaluate private deployment once the business scenario is confirmed. Downloading weights for self-hosting is worth the investment only when call volume is large enough, data compliance requirements are strict enough, and the team can operate inference services.

The value of open-weight models is that qualified teams gain a privately hostable, deeply customizable option. But it is not a free lunch—license terms, inference costs, and operational complexity together determine the real barrier to entry. Doing the math first matters more than jumping straight into deployment.