Open-Source LLM Local Deployment: Performance, Cost, and Limitations

    This article reviews a newly released family of open-source LLMs for local deployment, focusing on the 27B and 35B (3B active) variants. It examines reported speed and VRAM figures, quantization trade-offs, overly long reasoning, long-context interference, cost considerations, and when local deployment makes sense compared with cloud APIs. The goal is to help teams evaluate quality, latency, memory use, and operational cost with their own tasks rather than relying only on parameter counts or vendor benchmarks.

    What These Models Are

    A Chinese cloud provider recently released a series of open-source LLMs. The first public version uses a mixture-of-experts (MoE) architecture with 397 billion total parameters and 17 billion activated per inference. The vendor says its overall capability is close to its own cloud flagship closed-source model. This is a vendor claim, and independent evaluation needs to be checked separately.

    The series includes image-understanding capability from the design stage rather than as a later add-on. For developers, this means the same model can process both text and image inputs.

    Measured Performance of Mid-Size Versions

    The 27B-parameter version is the main option for local deployment. Early testers gave positive feedback on tasks such as code completion, bug fixing, unit test generation, multi-turn dialogue, and instruction following. For image understanding, the original article says image resolution is not high, but descriptions and analysis logic for image content are generally correct.

    The original article mentions that this version comes close to Sonnet 4.6 on some agent benchmarks. That result does not include a public evaluation set, sample size, or sampling parameters, so it only indicates that the model is close on the related tests. It cannot be used to conclude that the model is equivalent to closed-source models on all tasks.

    The 35B-parameter version (3B active) has weaker absolute capability but fast inference, making it suitable for latency-sensitive, high-frequency interaction scenarios such as input completion or simple question answering.

    The table below can help with selection, but its content is mainly based on the qualitative descriptions in the original article:

    Version Strengths Weaknesses Suitable Scenarios
    27B parameters Relatively comprehensive code, dialogue, and image understanding Slower, with longer thinking processes Local coding assistance, knowledge Q&A
    35B (3B active) Fast inference Weaker complex reasoning High-frequency interaction, low-latency applications

    Local Deployment Speed and VRAM: Numbers Depend on Test Conditions

    The original article provides several speed figures, but they come from informal community tests. The original does not give complete test conditions, so they cannot be reproduced.

    • 27B parameters, Q4 quantization, Q8 KV cache, Vulkan backend, single-GPU environment: about 23 tokens/s; the original says the context window can reach 100K.
    • 35B parameters (3B active), Q3 quantization, Q8 KV cache: about 97 tokens/s; the original says the context window can reach 256K.
    • In a CUDA environment, the 27B-parameter version runs at about 25 tokens/s, with total throughput of about 40 tokens/s under two concurrent requests.

    The test hardware in the original article was a modified RTX 2080 Ti with 22GB of VRAM. The original does not state the driver version, inference framework version, batch size, input prompt length, or output length, all of which can significantly affect speed. Longer prompts increase time to first token, so real-world perceived performance may differ greatly from these numbers.

    Context length also needs to be calculated separately. Model weights, KV cache, framework overhead, and system headroom all consume VRAM. On a 22GB GPU, a 100K or 256K context is very likely to exceed available space and must be verified by actual loading. A simple estimation method is to add weight VRAM, KV cache VRAM, framework overhead, and 10% to 20% headroom, then compare the result with the GPU’s available memory. This ratio is an empirical suggestion, not a fixed standard.

    MTP (multi-token prediction) is a newly added acceleration feature in inference frameworks. It can improve throughput, but the speedup depends on the model and framework version, and gains vary under different conditions.

    On cost, the original article says the solution requires a GPU in roughly the CNY 2,000 class. GPU prices vary by market and secondhand channels, so current prices should be verified before purchase, along with electricity, cooling, and chassis space.

    Quantization and Quality Loss

    Quantization formats such as Q4 and Q3 compress model weights and reduce VRAM usage. The trade-off is that output quality may decline. The original article notes that some quantized versions show instruction-following regression, such as tool-call format errors or unstable output structure.

    When choosing a quantization level, it is advisable to run comparative tests on your own task set, covering the scenarios you actually care about, such as code modification, structured output, and long-document Q&A. Choosing a quantization level based only on speed numbers can easily lead to a version that cannot meet requirements.

    Overly Long Reasoning

    The original article points out that the model produces lengthy thinking on simple brainteasers. The “car wash problem” and the “stick through the door problem” mentioned in the article sometimes lead to thinking processes exceeding 5,000 tokens. Even after arriving at the correct answer, the model may repeatedly verify it more than ten times. The original does not state the number of tests or the statistical method, so these numbers should be treated only as anecdotal descriptions.

    This overthinking slows response times and increases compute cost. Common mitigation directions include:

    • Adjust decoding parameters such as temperature, top_p, and repetition penalty. The original mentions that repetition penalty needs to be increased, but the specific value depends on the model version and sampling strategy, and there is no universal value.
    • Set a maximum output length or thinking-length cap and truncate beyond it.
    • Validate the format of tool calls and structured output, and retry automatically on failure.
    • Validate parameter combinations on a small test set first, then roll them out to production use.

    Increasing repetition penalty also has side effects. In tasks that require repeated words or phrases, such as repeated structures in code or verbatim quotes from source text, model performance may degrade. Parameter tuning needs to be evaluated together with the specific task.

    Long Context and History Interference

    The original article notes that in very long contexts, the model is sometimes distracted by irrelevant information in the conversation history. For applications that require long-term memory, it is not recommended to put all history directly into the context. Summarization, retrieval, or chunked processing can be used to keep only content relevant to the current question. These approaches need to be designed for the specific application; the model itself does not provide such capability.

    Impact on Cloud Providers: Possible, but Not Overnight

    The original article argues that improvements in local models will reduce the value of some general cloud APIs and push cloud services toward tiering. Sensitive data is not suitable for uploading to the cloud, and local deployment has practical significance for healthcare, finance, and internal enterprise code.

    These judgments have some merit but need qualifications. Local deployment does not mean zero cost; it requires hardware investment, operations staff, model updates, and security management. For enterprises with high concurrency and a need for stable service, cloud services still have clear advantages. Tasks that local models can handle may not reach the quality of the top models.

    Selection Recommendations

    Deployment location can be divided by three task types:

    1. Edge devices: speech transcription, meeting summaries, and simple text polishing. These require low latency and strong privacy, and task complexity is relatively low.
    2. Private servers or edge nodes: enterprise knowledge base Q&A, code review, and internal agents. Data does not leave the premises, and medium compute is required.
    3. Central cloud: tasks that require a large knowledge base, complex reasoning, or the highest-quality output.

    When selecting a deployment option, first test quality, speed, and VRAM usage on actual tasks, then decide where to deploy. Do not look only at parameter counts or vendor-published benchmark scores.