Lightweight LLM Selection for Office Work: A Reusable Evaluation Framework

    This article presents a reusable framework for evaluating lightweight LLMs on office tasks. It describes six representative tasks, a scoring and logging scheme, how to interpret sample results, cost estimation, task-based model selection, and a hybrid routing, caching, and fallback architecture. It also covers limitations and implementation steps, with hypothetical sample data used only to illustrate the evaluation method.

    Why Evaluate Lightweight Models Separately?

    Large language models are increasingly splitting into two tiers. Flagship models are good at complex reasoning and long-form generation, while lightweight models use smaller parameter counts or knowledge distillation to achieve lower invocation costs and shorter first-token latency. High-frequency office tasks such as reading documents, drafting proposals, scheduling, organizing spreadsheets, generating reports, and revising requirements usually have clear structures and fixed formats. They are well suited to testing whether a lightweight model is sufficient.

    However, cost-effectiveness cannot be judged by impression. A model may be fast at single-step text generation but may not perform well on tasks that require multiple tool calls. Before selection, teams need to break tasks down and record results item by item.

    Designing Six Office Task Categories

    To reduce subjective bias, use the same fictional background, such as “annual AI conference planning.” Set the conference size to 350 attendees, a budget cap of RMB 300,000, venue capacity of 400, and different guest availability. All models use the same prompt template and the same document-reading and spreadsheet tools.

    The six subtasks are:

    1. Material understanding: extract hard constraints from old proposals, quotations, and guest availability documents.
    2. Proposal writing: generate a first draft covering the theme, agenda, guest strategy, and budget allocation.
    3. Schedule planning: produce conflict-free speaking slots and handle cases where multiple guests are available in the same slot.
    4. Spreadsheet processing: output a budget breakdown, calculate the share of each category, and calculate the remaining budget.
    5. Report generation: compress the proposal into an outline of no more than 10 pages and provide chart suggestions.
    6. Requirement revision: make incremental adjustments under conditions such as a 20% budget cut and a half-day reduction in the event duration, and explain what changed.

    Each task must deliver usable files or operation results, not just an approach.

    Scoring and Logging

    For each task, record four types of data: completion time, output word count, format compliance rate, and quality score. The quality score uses a 1-to-5 scale, and the scoring rubric must be written in advance, for example:

    • 5: All constraints are met, there are no numerical errors, and the output can be used directly.
    • 3: Constraints are basically met, but one manual correction is needed.
    • 1: Key constraints are missing, or the output cannot be used.

    Use at least two raters, each scoring independently. If the difference between the two scores exceeds 1 point, a third person reviews the result. The more samples, the more stable the conclusion. The example in this article uses only one fictional scenario and can only illustrate the method.

    Each invocation should also record the model version, call date, prompt version, tool version, and raw logs. Without this information, results cannot be reproduced.

    How to Read Sample Results

    The table below contains hypothetical demonstration data to show how to read comparison results. It does not represent the performance of any real product.

    Metric Model A Model B Model C
    Material understanding time 8 s 12 s 10 s
    Proposal drafting time (~1,500 words) 25 s 23 s 27 s
    Tool call timeouts 0 0 1
    Budget share error 0.00% 0.5% 2.3%

    When reading this type of data, look at constraint-related tasks first. Suppose a model interprets “budget cap” as “suggested budget”; later proposals will then exceed the limit. This type of error has a much larger impact than spending a few extra seconds.

    Next, look at the number of tool calls. More calls increase end-to-end time and token consumption. Fast single-step reasoning does not mean the entire task is fast.

    Finally, look at revision tasks. A good model should change only the affected parts and explain what was changed, rather than rewriting the entire document.

    How to Calculate Cost

    Cost is usually estimated per 1,000 tasks. The cost of a single task equals the input token count and output token count multiplied by their respective unit prices, plus the fee for each tool call. Retries must also be included.

    When comparing two models, state the assumptions clearly: average number of calls per task, average input and output lengths, whether retries are included, and the pricing date. Different assumptions lead to different conclusions.

    For pure text generation tasks, the cost gap between two models is often small. For multi-step tasks such as material understanding plus spreadsheet processing, models with fewer tool calls are often cheaper. Therefore, cost comparison must be considered together with task structure.

    Selecting Models by Task Type

    • Daily writing tasks, such as meeting notes, weekly reports, and request forms: prioritize response speed and unit price. Flagship-level reasoning is usually unnecessary.
    • Structured data tasks, such as financial summaries and shift schedules: prioritize tool-call stability and numerical accuracy. Key numbers should be sampled and checked.
    • Iterative revision tasks, such as proposal reviews and requirement changes: prioritize incremental modification capability, meaning whether the model can change only the affected paragraphs and explain the changes.

    Before selection, count the ten most frequently used tasks in the team, label each task type, and test with real samples.

    Hybrid Architecture: Routing, Caching, and Fallback

    Office agents usually do not call only one model. A common structure has three layers.

    1. Routing layer: distribute requests based on task complexity. The basis can be keywords, input token count, or estimated number of steps. For example, single-step writing can be sent to a lightweight model, while tasks involving multiple files and multiple calculations can be sent to a flagship model.
    2. Caching layer: for high-frequency requests with fixed answers, such as leave approval and meeting-room lookup, use template replies directly without calling a model.
    3. Fallback strategy: when a lightweight model outputs results below the quality threshold twice in a row, automatically hand the task to a flagship model for a retry. The threshold needs to be calibrated with human-rated samples.

    The benefit of this structure is that simple tasks keep low latency and low cost, while complex tasks still have sufficient capability as a backstop. The cost is that routing rules and quality monitoring must be maintained. If fallback is triggered too frequently, total cost may rise instead.

    Limitations of the Evaluation

    The method in this article has several clear limitations:

    • The tasks focus on document processing and do not cover multimodal needs such as voice and images.
    • Only a single fictional scenario is tested; memory performance in long multi-turn conversations is not simulated.
    • Cost includes only API fees, not the hardware and operations costs of private deployment.
    • The sample data is hypothetical demonstration data and cannot be generalized into capability conclusions about any real model.

    If evaluation results are to be used for formal decisions, additional information is needed, including model versions, call dates, full prompts, toolchain versions, and raw logs.

    Implementation Steps

    1. List the team’s high-frequency office tasks and take the top ten.
    2. Prepare 10 to 20 real samples for each task type and remove sensitive information.
    3. Have two raters score independently according to the rubric in this article.
    4. Record the model version, date, latency, and token usage for each call.
    5. Set routing rules and fallback thresholds based on the results, and retest regularly.

    Whether a lightweight model can handle a given job depends on the task structure and the team’s acceptance criteria. Rather than looking only at public leaderboard rankings, it is better to run a repeatable test with your own tasks. Only then can the conclusions truly guide model selection.