Zhipu GLM-4.7 Hands-On: Coding Capability Advances and Reasoning Efficiency Analysis

    An observer recorded their hands-on experience with Zhipu's GLM-4.7, and this article distills and expands that into a neutral analysis. It revisits three core questions: how to scientifically evaluate a coding LLM, how to assess its reasoning efficiency, and how to turn evaluation results into real engineering decisions. You will find a study of generational capability progression, a breakdown of token efficiency, a five-dimension coding comparison against an industry benchmark, practical engineering techniques for closing stability gaps, a self-built evaluation methodology with scoring and cost-accounting scripts, and a deployment checklist for Baidu AI Cloud. It is written for developers who want to run a second, independent evaluation rather than take a leaderboard at face value.

    One observer documented their hands-on experience with Zhipu’s GLM-4.7, and this article builds on that with neutral distillation and deeper analysis. It returns to three core questions: how to scientifically evaluate a coding LLM, how to assess its reasoning efficiency, and how to make evaluation results actually serve engineering model selection. The article closes with a self-built evaluation set, a token cost-accounting script, and a Baidu AI Cloud integration checklist for developers who need to run their own second round of testing.

    1. Iteration watch: from “advancing on all fronts” to “focused general capability”

    Looking back at Zhipu’s product cadence over recent years, the observer summed it up in one phrase: steady and deliberate progress.

    • In 2024, its model lineup aimed squarely at the leading international vendors, advancing on every product line at once, yet the market response was fairly muted.
    • Starting in 2025, the strategy shifted toward focusing on general-purpose models.
    • Its domestic coding models (the GLM series) moved from a rough first generation, to a preliminary Agent positioning in 4.5, to establishing a leading domestic position in coding with 4.6, and finally to a concentrated payoff with 4.7.

    This trajectory carries two lessons for engineering teams.

    First, model capability evolves through generational stacking. From context length and hallucination control, to Agent capability, to coding specialization and reasoning efficiency, each generation usually has to polish several dimensions at once. The weak spots left by one generation often become the main focus of the next upgrade. So when selecting a model, rather than betting on a single “wow moment” from one release, it is more useful to watch whether the vendor shows clear, continuous problem-fixing ability across adjacent generations.

    Second, how tightly a product line is focused is worth watching too. When a product line spreads too wide, the iteration pace of any single capability tends to get diluted. Vendors with a clear focus usually iterate more densely and with a sharper positioning in a specific track such as coding.

    1.1 The key change: a capability leap, not minor patches

    The observer’s comparison: if 4.6 over 4.5 was just “minor patches,” then 4.7 can fairly be called a “transformation.” This shows up in two ways:

    • The relationship between reasoning and non-reasoning modes was redefined. The new generation, running in non-reasoning mode only, already matches the previous generation with deep reasoning switched on.
    • The reasoning mode enters the first tier in the coding niche, competing in the same range as the industry-recognized coding benchmark models.

    From an engineering standpoint, this directly affects cost structure. In the past, to get high accuracy on difficult tasks, teams often had to enable reasoning mode for every request and pay for the long chains of thought. Now tasks can be handled in two layers:

    • Routine coding, completion, and explanation tasks run in non-reasoning mode;
    • Hard tasks such as complex refactoring and tricky bug localization call on reasoning mode.

    This is exactly why many teams adopt a “dual-mode combination” when selecting a coding model today.

    2. Reasoning efficiency: why token efficiency matters as much as intelligence

    Beyond the capability gains themselves, this version also made a breakthrough in reasoning efficiency. In the observer’s tests, a single task in non-reasoning mode consumed only about 40% of the tokens of the previous generation.

    2.1 The two biggest sources of token consumption

    Many developers only care about a model’s raw “intelligence” and overlook that the tokens consumed per call are a direct source of both cost and latency. Token consumption comes mainly from three parts:

    1. Context: code files, requirement descriptions, conversation history;
    2. Input and output: the model’s understanding of the task and the code it generates;
    3. The chain of thought in reasoning mode: this can consume far more tokens than the final answer itself.

    For the same task success rate, if token consumption differs by more than 2x, the cost gap gets multiplied once you deploy at scale.

    2.2 Token efficiency is the core value of reasoning mode

    Token efficiency matters especially for reasoning mode, for a few reasons:

    • Reasoning mode trades a longer chain of thought for higher accuracy, so its token overhead is inherently higher;
    • Hard tasks often require multiple attempts, which multiplies the single-call overhead several times over;
    • If a single task routinely burns tens of thousands of tokens, the cost becomes hard to bear in batch scenarios.

    “Reaching the same or better results with fewer tokens” is precisely an important dimension of competition among reasoning models, and it is the direction some vendors with relatively constrained resources are concentrating their efforts.

    2.3 Building a two-dimensional “capability–cost” selection framework

    A more scientific approach is to compare capability evaluation and token cost accounting in the same table, rather than looking only at leaderboard rankings. At minimum, record the following fields:

    Field Meaning How to collect
    task_id Unique task identifier Fixed in the evaluation set
    model_mode Model and mode (non-reasoning / reasoning) Call parameters
    pass Whether it passed (unit tests pass or manual judgment) Automated scoring
    input_tokens Number of input tokens usage field in the response
    output_tokens Number of output tokens (including reasoning tokens) usage field in the response
    latency_ms End-to-end elapsed time Client-side timing
    cost Cost per call Token count × unit price

    When aggregating, a more meaningful metric than “average token count” is cost per effective output:

    1. cost per effective output = total spend / number of passed tasks

    This metric accounts for both accuracy and overhead at once. A model that is cheap per call may not actually cost less if it needs to be retried repeatedly. As for the claim that “token consumption is only 40% of the previous generation,” that comes from observation under a specific evaluation set and measurement method. When reproducing it, be sure to measure with your own set of tasks, and do not simply copy the number.

    3. Coding fundamentals: a five-dimension horizontal comparison

    For coding tasks, the observer also ran a comparison against industry benchmarks across several dimensions.

    On basic coding ability, GLM-4.7 significantly outperformed GLM-4.6 on nearly every question type, so a generational comparison was no longer meaningful. The focus therefore shifted to GLM-4.7 (Think) against the benchmark Sonnet 4.5 (Think), which scored closest.

    Advantage 1: Context hallucination suppression. On long-text information extraction checks, 4.7’s reasoning mode suppressed hallucinations better and was more likely to produce low-hallucination results.

    • Taking log information parsing as an example, the benchmark model made mistakes more often and could amplify numerical hallucinations;
    • 4.7, by contrast, performed reliably.

    That said, 4.7’s hallucination rate is not the lowest, and it still trails the world’s first tier. For instance, on the “target number” question (#43), it occasionally misread the problem and used a number that does not appear in it.

    Advantage 2: Long-chain reasoning. On tasks that need more steps and longer chains of reasoning, 4.7 delivered better peak performance.

    • The benchmark model’s reasoning-mode intelligence was not outstanding among top models, though it qualifies as a “gatekeeper of the first tier”;
    • 4.7 has already “got one foot through the door.”

    The reason it is only one foot is that it can lose a few points on minor issues across different problems, making it hard to reliably earn a perfect score.

    Advantage 3: Mathematical calculation. Math problems may be an area the benchmark model deliberately gave up on, or at least did not specifically train for. Building on the big improvement 4.6 made in math, 4.7 further largely solved the calculation precision problem, and direct-computation questions could be answered with high precision and no error.

    Advantage 4: Instruction following. 4.7’s instruction-following ability was basically no weaker than the benchmark model’s:

    • It handled conditional instructions (such as “if … then …”) slightly better;
    • The benchmark model was more stable on direct instructions.

    Overall, the two are at the same level.

    3.1 Why leaderboards should be read “with footnotes”

    It is worth noting that these comparison tables usually carry a note such as “only shows a partial set of comparable models, not a complete ranking,” along with the source of the questions (for example, questions 53 and 54 newly added to a horizontal review’s November leaderboard) and the leaderboard homepage. This reminds us:

    • Leaderboard rankings depend on the question set, and changes in the proportion of new and hard questions affect placements;
    • The comparison tables vendors or evaluators show are often “curated,” presenting their strong points;
    • The same model can rank differently under different evaluation methods.

    So leaderboards are suitable as an initial screening reference, but the final decision should return to a small-scale hands-on test in your own business scenario.

    4. Deeper interpretation: how engineering teams should use these conclusions

    4.1 Applying “hallucination suppression” to long-document parsing

    Tasks like log parsing, report extraction, and contract clause comparison are essentially “long context + precise localization” tasks that are extremely sensitive to hallucinations. It helps to make the following requirements explicit in the prompt:

    • Answer only based on the given content; do not infer or fill in missing fields;
    • Output structured results (such as JSON), explicitly marking missing fields as null;
    • For key numbers, require quoting the original text fragment so humans can verify.

    Example prompt:

    1. You are a log analysis assistant. Answer questions based only on the [LOG CONTENT] below; do not use any external knowledge or make inferences.
    2. The output format is JSON, with the following field descriptions:
    3. - level: log level, one of [ERROR, WARN, INFO]
    4. - error_code: error code; if it does not appear in the log, use null
    5. - evidence: the original text fragment supporting the above judgment, quoted verbatim
    6. [LOG CONTENT]
    7. {{log_content}}

    At the same time, add a second validation pass on the model output at the code level, for example checking that error_code actually appears in the original text:

    1. import json
    2. def parse_log_with_model(model_reply: str, raw_log: str) -> dict:
    3. result = json.loads(model_reply)
    4. code = result.get("error_code")
    5. # Hallucination check: the error code the model returns must be found verbatim in the raw log
    6. if code is not None and code not in raw_log:
    7. result["error_code"] = None
    8. result["_warning"] = "The error code returned by the model was not found in the raw log; it has been set to null pending manual review"
    9. return result

    The benefit is that even if the model hallucinates occasionally, the business layer has a fallback validation mechanism, so incorrect data is not passed directly to downstream systems.

    4.2 Long-chain reasoning: improve stability with “multiple sampling + self-checking”

    “Got one foot through the door” means the model’s peak capability is already strong, but its stability still needs engineering support. For a model whose single-pass success rate has not yet reached a perfect score, common approaches include:

    • Best-of-N sampling: generate multiple candidates for the same task, then pick the best with tests or a verifier;
    • Add a self-check step: have the model review key steps (such as boundary conditions and variable definitions) before giving its answer;
    • Structured output constraints: require the model to list steps first and give the conclusion second, making it easier to locate where an error occurred.

    Illustrative code (pseudocode):

    1. def solve_with_self_check(prompt: str, model, verifier, k: int = 3):
    2. candidates = [model.generate(prompt, temperature=0.7) for _ in range(k)]
    3. # For each candidate, have the model self-check once, then validate with the verifier
    4. checked = []
    5. for ans in candidates:
    6. reviewed = model.generate(f"Please check whether the following solution has any errors:\n{ans}")
    7. if verifier(reviewed, ans):
    8. checked.append(ans)
    9. return checked[0] if checked else None

    In engineering practice, you can use the test pass rate as the verifier and “whether the final answer agrees with the unit test result” as the selection criterion, which is especially effective for coding tasks.

    4.3 Math and numeric computation: do not let the model “do mental math” on key numbers

    Even with the significant improvement in math ability, for business logic involving amounts, inventory, and metric calculations, it is still advisable to adopt a division of labor in which “the model handles intent understanding and the tool handles precise calculation”:

    • The model parses the requirement and generates a calculation expression or calls a calculation tool;
    • A deterministic program (such as a calculator or script) performs the numeric computation;
    • The result is validated for legality of units, magnitude, and precision.

    This “model + tool” combination can bring the risk of numeric errors close to zero.

    4.4 Instruction following: distinguish “direct instructions” from “conditional instructions”

    Different models have different sensitivities to instruction types. It helps, in prompt engineering, to:

    • For direct instructions, use short, clear imperatives and avoid ambiguity;
    • For conditional instructions, write out the judgment condition and the corresponding action explicitly, splitting into multiple rules when necessary;
    • Constrain the format with structured output (a JSON Schema) to reduce parse failures.
    1. {
    2. "type": "object",
    3. "properties": {
    4. "action": {"type": "string", "enum": ["approve", "reject", "manual_review"]},
    5. "reason": {"type": "string"}
    6. },
    7. "required": ["action", "reason"]
    8. }

    5. Building your own evaluation set: a more reliable selection method than “reading leaderboards”

    5.1 Design principles for the evaluation set

    • Close to the business: draw questions from your team’s real code repositories and production issues, not public question banks;
    • Layered coverage: include a mix of simple completion, medium refactoring, hard bug localization, and long-context understanding;
    • Contamination resistance: refresh questions regularly to avoid inflation from questions that have entered public training data;
    • Automated scoring: judge coding questions with unit tests where possible to reduce manual effort.

    5.2 A minimal, runnable scoring script

    1. import json, time
    2. def evaluate(model_client, cases, unit_price_per_1k):
    3. rows = []
    4. for c in cases:
    5. t0 = time.time()
    6. resp = model_client.chat(c["prompt"])
    7. latency = (time.time() - t0) * 1000
    8. usage = resp.get("usage", {})
    9. rows.append({
    10. "task_id": c["id"],
    11. "pass": run_tests(resp["content"], c["tests"]),
    12. "input_tokens": usage.get("prompt_tokens", 0),
    13. "output_tokens": usage.get("completion_tokens", 0),
    14. "latency_ms": round(latency),
    15. })
    16. total_cost = sum((r["input_tokens"] + r["output_tokens"]) / 1000 * unit_price_per_1k for r in rows)
    17. passed = sum(r["pass"] for r in rows)
    18. report = {
    19. "pass_rate": passed / len(rows),
    20. "avg_latency_ms": sum(r["latency_ms"] for r in rows) / len(rows),
    21. "cost_per_pass": total_cost / passed if passed else None,
    22. }
    23. return report, rows

    As long as you swap out model_client and the question set cases, this script can be reused for any model and any mode, allowing horizontal comparison across the three dimensions of capability, latency, and cost.

    5.3 Regression and version management

    Model vendors keep iterating. The version that performs best today may be surpassed tomorrow, and localized regressions can also appear. It is advisable to:

    • Fix the evaluation set and scoring script under version control;
    • Run a regression each time a model is upgraded or a vendor is switched, to observe whether any capability has regressed;
    • Record the results of each evaluation round to build a traceable selection archive.

    6. A deployment checklist on Baidu AI Cloud

    When evaluation conclusions land in a production environment, they usually involve the following general capability modules. For specific capabilities and how to enable them, refer to Baidu AI Cloud’s official documentation:

    • Model service: integrate candidate models via API, with support for switching between non-reasoning and reasoning modes, billed by token;
    • Object storage: store the evaluation set, log samples, generated results, and evaluation reports for easy version traceability;
    • Message queue: smooth out traffic peaks and handle asynchronous tasks such as batch code generation and batch log parsing;
    • Container platform: deploy custom verifiers, unit-test execution sandboxes, and other evaluation components, with elastic scaling;
    • Logging and monitoring/alerting: collect metrics such as call volume, token consumption, latency, and pass rate, and configure threshold alerts to catch capability regressions or cost anomalies early;
    • Data management tools: perform masking and access control on the evaluation set and production samples to meet compliance requirements.

    7. Summary and outlook

    From 4.6 to 4.7, the change in this generation is not merely “a little stronger,” but rather:

    • The non-reasoning mode matches the previous generation’s reasoning mode;
    • The reasoning mode enters the first tier in the coding track;
    • Token efficiency improved substantially.

    For developers, what is truly valuable is not memorizing a model’s ranking, but mastering a methodology of “evaluate–account for cost–deploy”:

    • Replace a single leaderboard with your own evaluation set;
    • Use the two-dimensional “capability–cost” framework to make selections;
    • Use engineering means (validation, self-checking, tool calls) to shore up the model’s stability gaps.

    Looking ahead, as reasoning efficiency keeps improving and Agent capability keeps strengthening, the focus of competition among coding LLMs will gradually shift from “who is smarter” to “who is more stable, cheaper, and easier to use.” Building an evaluation and cost-accounting system early is what will let you make the right technical decisions quickly in the next round of iteration.