Open-Source LLMs with Vision and Computer Use: Selection Criteria for Production

    Open-source large language models are expanding beyond parameter scale and benchmark scores into visual understanding and computer operation. This article explains what these capabilities can solve, why open-source deployment matters for sensitive industries, how to interpret hallucination-rate data and pricing estimates, and what security controls to put in place before integration. It also provides a selection checklist covering data sensitivity, complex reasoning, long context, multimodality, computer use, cost, and compliance audits, with a recommended approach of starting from small, real-world pilots.

    The capability boundary of LLMs is expanding

    Over the past two years, competition among open-source large models has focused mainly on parameter scale and benchmark scores. Once parameter counts reach very high levels, the returns from continuing to stack more parameters become noticeably smaller. A new generation of open-source models is shifting emphasis to two capabilities: visual understanding and computer operation. The former allows a model to read information from charts, documents, and images. The latter allows a model to complete multi-step tasks through a graphical interface or command line. This means models are gradually moving from answering questions to executing tasks.

    What vision and computer-use capabilities can solve

    Many real workflows are not a single question-and-answer exchange. For example, a workflow may involve opening a PDF report, extracting data from it, generating a chart, running an analysis script, and finally switching between multiple software applications. Previous models could write code and answer questions, but they often failed at the handoffs between steps. A model that can understand charts and operate interfaces can connect these steps.

    However, the actual effectiveness of these capabilities depends on task type, interface complexity, and failure rate. Successful cases in demo videos do not represent stable performance. Teams should therefore run small-scale tests on their own real tasks and record completion rates and the proportion of cases requiring human intervention.

    Why open source matters for sensitive industries

    The value of open source lies in the ability for enterprises to deploy, fine-tune, and audit models independently. Industries such as finance, healthcare, and government have high data-compliance requirements. Calling third-party black-box APIs means data leaving the organization and dependence on external services, while local deployment makes it easier to meet internal security policies.

    Some open-source models commit to publishing weights and providing model system cards. Teams can use these to understand training and evaluation information and adapt the model to domain knowledge. It is important to note that open source does not automatically mean secure or reliable. Before use, teams still need to review license terms and confirm the scope of commercial use.

    How to interpret hallucination-rate data

    Hallucination rate is a key metric for determining whether a model can enter production workflows. The original report mentioned that one model’s hallucination rate in a financial-task-related evaluation dropped from 15.76% to 5.45%. This figure has reference value, but readers should first check the evaluation set, sample size, model version, and measurement methodology in the original report. The same model can perform very differently across different evaluation sets.

    For automated decision-making and externally published content generation, it is advisable to retain a human review step. Do not remove review simply because of a single metric.

    Pricing: calculate carefully before deciding

    Price is an important factor affecting large-scale adoption. According to the report, one open-source model’s input price was approximately RMB 6 per million tokens and output approximately RMB 18, with lower prices when cache hits occur, and some cloud service providers were also offering promotional discounts. The currency, applicable region, tax inclusion, cache-hit conditions, and promotional period for these prices should be confirmed against the provider’s current announcement.

    Consider a hypothetical scenario. A business processes 1 million tokens per day, with input and output each accounting for half. Based on the prices above, the daily cost would be approximately RMB 12, and the monthly cost approximately RMB 360. This estimate includes only model invocation costs and does not include development, API calls, monitoring, or human review. To compare with labor costs, use the organization’s actual working hours and unit rates; do not directly apply a fixed multiplier.

    From tool to colleague: security design before integration

    For ordinary users, the most direct value is handing repetitive cognitive work to a model. Companion agents can connect to email, documents, calendars, and meeting notes to help summarize project progress, identify to-dos, and remind users of deadlines. In design, development, and content creation, models can also generate drafts, adjust parameters, or output web prototypes.

    Models that can operate systems also carry higher risk. Before integrating them into a production environment, consider the following principles:

    • Least privilege: grant only the directories, applications, and account permissions required to complete the task.
    • Sandbox execution: run tasks first in an isolated environment, and open access to real systems only after confirming normal behavior.
    • Sensitive-data redaction: remove information such as ID numbers, bank card numbers, and medical records from content sent to the model.
    • Operation auditing: record each instruction, command, and execution result for later troubleshooting.
    • Human confirmation: operations such as deletion, sending, payment, and permission changes must be confirmed by a human.
    • Rollback capability: back up before critical operations to ensure recovery if something goes wrong.

    Selection dimensions to compare

    No single model leads in every dimension. The table below lists the dimensions to check during model selection.

    Dimension Questions to confirm Better-fit scenarios
    Data sensitivity Can data leave the organization, and is local deployment required? Internal systems in finance, healthcare, and government
    Complex reasoning How does the model perform on long-chain tasks in the domain? R&D and analysis requiring rigorous derivation
    Long context What are the recall rate and consistency of information across long documents? Contract, financial report, and technical document processing
    Multimodality How accurate is recognition of charts and scanned documents? Report extraction and document digitization
    Computer use What is the task completion rate, and how are failures handled? Repetitive work with clearly defined steps
    Cost What is the total cost per task, including development and operations? High-frequency, standardized business calls
    Compliance auditing What licenses, model cards, and logging capabilities are available? Industries requiring traceability and review

    Closed-source flagship models still have advantages in extremely complex tasks and scenarios with abundant compute. Open-source models are easier to adopt in scenarios requiring data control, deep customization, and cost sensitivity. The two are not necessarily an either-or choice. Many teams split tasks by type, assigning sensitive tasks to local models and high-difficulty tasks to cloud services.

    Conclusion: start with a small pilot

    Progress in visual understanding, task execution, and cost is giving more teams the opportunity to try frontier capabilities. But improved capability does not mean a model can directly replace human workers. A safer approach is to select one or two high-frequency, low-risk tasks, test accuracy, completion rate, and cost with real data, and then decide whether to expand. Only by getting evaluation, permissions, and auditing right can a model become a trustworthy digital colleague.