Evaluating Small and Medium-Sized LLMs: Capabilities and Practical Observations

I. Background: Why Focus on Small and Medium-Sized LLMs?

As Large Language Model (LLM) technology rapidly evolves, the industry’s focus has shifted from “the larger the parameter size, the better” to “achieving stable capabilities at controllable costs.” Especially in scenarios such as enterprise applications, edge deployment, R&D for small teams, and automated testing, small and medium-sized models are becoming a more practical choice.

These models typically share the following characteristics:

1. Relatively low VRAM consumption: Easier to run on a single GPU or in lightweight inference environments.
2. Low inference latency: Suitable for interactive applications, batch processing, and tool calling.
3. Controllable training and fine-tuning costs: Teams can perform continued training with limited resources based on their own data.
4. Clear capability boundaries: Better suited for embedding into fixed workflows rather than acting as open-domain super assistants.

Therefore, when evaluating a small or medium-sized model, one should not only look at absolute scores but also comprehensively judge it based on task types, inference “feel,” stability, and integration difficulty. Based on an internal technical evaluation, this article discusses the performance differences of these models across multiple benchmarks and their experience characteristics in practical use.

II. Evaluation Methodology: Multi-Benchmark Comparison and Capability Profiling

In this evaluation, we selected several small and medium-sized models of the same class as references. It should be noted that “same class” primarily refers to similar parameter scales, comparable deployment profiles, and consistent target scenarios, rather than simple brand comparisons. The goal was not to crown the “strongest model,” but to determine whether a specific model could form an advantage in specific engineering scenarios.

2.1 Benchmark Dimensions

The evaluation covered 28 common capability dimensions, including but not limited to:

• Common sense reasoning
• Mathematical operations
• Code generation and completion
• Instruction following
• Multi-turn dialogue consistency
• Text summarization
• Information extraction
• Logical judgment
• Safety and compliance responses
• Tool calling comprehension

These capabilities were categorized into four levels: input understanding, reasoning generation, interaction stability, and engineering usability.

2.2 Comparison Overview

The results showed that the target model outperformed Model A in 27 out of 28 dimensions and held a high proportion of leads over Model B. However, when compared to the larger-scale Model C, it fell short in 17 dimensions. This outcome was not surprising.

The reasons are:

• Small models perform well in single-turn knowledge QA, basic code completion, and structured output.
• However, in complex multi-hop reasoning, open-domain long-text generation, and ambiguous instruction following, the advantages of a larger parameter scale remain evident.
• Model C exhibits a stronger “large model feel,” particularly in organizing long answers, tonal naturalness, and context coverage, making it closer to large-scale conversational systems.

Therefore, the evaluation conclusion should be stated as: Among similar small and medium-sized models, the target model performs notably well; however, a natural gap remains when compared to larger-scale models.

III. Model Capability Analysis: Strengths and Weaknesses

3.1 Strength Scenarios

The main advantages of the target model are reflected in the following areas.

3.1.1 Code and Structured Tasks

In tasks such as code completion, function generation, error localization, and formatted output, the target model demonstrated high stability. For instance, given a function signature and comments, it could generate structurally complete code snippets with standard naming conventions.

# Input: Calculate the total amount from an order list and filter invalid orders
def calculate_total(orders):
    """
    orders: list[dict], each order contains amount and status fields
    """
    total = 0
    for order in orders:
        if order.get("status") == "valid":
            total += order.get("amount", 0)
    return total

These tasks require high structural understanding from the model but rely less on complex background knowledge, making small models perform admirably.

3.1.2 Information Extraction and Format Control

When extracting fields from unstructured text and generating JSON output, the target model adhered well to field constraints and output formats. During the evaluation, we used 50 sample sets requiring a stable output of a specific structure:

{
  "entity": "",
  "relation": "",
  "confidence": 0.0
}

The target model exceeded same-scale reference models in both field completion rate and format legality rate, which is crucial for subsequent pipeline-based development.

3.1.3 Instruction Following and Short Responses

In tasks like short Q&A, classification, selection, and rewriting, the target model’s output was more concise and less prone to invalid expansion. For API-style calls requiring low latency and high throughput, this characteristic significantly reduces post-processing costs.

3.2 Weakness Scenarios

3.2.1 Long-Text Generation and Complex Organization

When required to generate long technical proposals, industry analyses, or cross-paragraph arguments, the target model occasionally exhibited local repetition, logical jumps, or loose structures. In contrast, larger models were more natural in paragraph transitions, highlighting key points, and maintaining tonal consistency.

3.2.2 Ambiguous Intent Processing

For highly ambiguous questions like “help me look at this requirement” or “how is this solution,” the target model sometimes relied too heavily on surface-level keywords, lacking the ability to proactively ask follow-up questions or clarify. This affects the user experience in conversational products.

3.2.3 Multi-Turn State Tracking

In multi-turn dialogues with extended context, the target model occasionally lost early constraints. For example, if a user requested “keep answers under 200 words” in the first turn, the model might ignore this limit when asked a question again in the fifth turn.

IV. Understanding the “Large Model Feel”

The “large model feel” is not a strict academic metric, but it is crucial in practical use. It typically encompasses:

• Whether the response is natural;
• Whether it can understand subtext;
• Whether long texts are well-organized;
• Whether it knows when to explain and when to directly provide a conclusion;
• Whether it proactively explains when facing uncertain issues.

These experiential differences are hard to reflect through a single benchmark score. A model might score high on multiple benchmarks but still feel “rigid” or “mechanical” during actual chat. Therefore, during technical selection, it is recommended to incorporate human review, especially blind testing against high-frequency real-world requests.

V. Practical Deployment Recommendations

5.1 Prioritize Scenario Matching

Do not pursue the “absolute strongest overall.” Instead, select the model based on business scenarios. For example:

• For internal knowledge base Q&A, focus on evaluating information extraction and short-response capabilities.
• For coding assistants, prioritize testing code completion, explanation, and error localization.
• For customer service bots, multi-turn tracking, safety compliance, and tonal naturalness are more critical.

5.2 Establish a Tiered Model Strategy

In complex systems, a tiered strategy of “small models for high-frequency tasks, large models as fallback for complex requests” can be adopted. For example:

1. Use a small model to classify the question type.
2. Directly return results for structured tasks the small model can handle.
3. Route to a large model when the small model’s confidence is insufficient.

This balances overall performance and cost.

5.3 Focus on Inference Deployment Costs

One advantage of small and medium models is deployment flexibility. During evaluation, alongside accuracy, you should also record:

• First token latency;
• Average generation speed;
• VRAM consumption;
• Long-term running stability;
• Dynamic batching efficiency.

These metrics often impact the live user experience more than individual benchmark scores.

VI. Conclusion

In this evaluation, a small/medium-sized model with a parameter scale of less than 1T outperformed same-class reference objects in most capability dimensions. This indicates substantial effort in engineering optimization, training data quality, and task coverage. However, compared to larger-scale models, a gap remains in the comprehensive “feel” for complex scenarios.

This gap is not a defect but a result of the model’s positioning. The value of small and medium-sized models lies in being “sufficient, fast, and cost-effective.” If leveraged in appropriate scenarios and backed up by larger models, they are fully capable of serving as core components in production systems. Therefore, in practical selection, we should focus more on whether the model matches our business boundaries rather than pursuing absolute leadership on a specific leaderboard.