🔍 Read the full analysis: Why The Benchmark Ranking Favors Z.ai’s Legal AI Despite Its 1-In-5 Pass Rate on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Vals AI’s Harvey Legal Agent Benchmark leaderboard, placing seventh among 81 models in the archived results. The score does not mean it met only 19.17% of individual criteria: Vals reports a 92.5% criteria pass rate, while the task score requires every criterion to pass. The result is a benchmark ranking, not evidence that the model can be used without legal review.
Z.ai’s GLM-5.3-FlashX-Legal placed seventh among 81 models on Vals AI’s archived Harvey Legal Agent Benchmark leaderboard, despite passing every required criterion on just 19.17% of tasks. The apparent contradiction comes from the benchmark’s all-or-nothing scoring: a task counts as complete only when every item in its rubric passes, while Vals reports that the model passed 92.5% of individual criteria.
The result is listed in Vals AI’s archived Version 1 leaderboard, last updated October 18, 2026. Vals describes its evaluation as using a 120-task private holdout set and Harvey’s grading protocol. Two language-model judges grade each submission, and the reported task score averages their pass rates. The supplied documentation says internet access is disabled during the evaluation.
Z.ai’s October 13 announcement says the legal model improved on the underlying GLM-5.3-Flash base model by 12.5 percentage points. The archived table supports that arithmetic: GLM Legal scored 19.17%, compared with 6.67% for the base model. The company also describes the model as state of the art among open-source models in the evaluation, but the cited material does not specify the comparison set, and the source report says it found no published weights for the legal model.
The leaderboard lists GLM Legal’s cost at $0.15 per test, compared with $0.57 for the base model. Z.ai separately lists API pricing of $0.37 per million input tokens and $1.25 per million output tokens. These are different cost measures: the leaderboard’s per-test figure should not be treated as the model’s general API price or as a complete estimate of deployment costs.
What the Score Means for Legal Work
The ranking shows that task completion and item-level accuracy are different measures. A model can satisfy most individual requirements while still fail a complete legal assignment if one required deadline, citation, conclusion or formatting instruction is wrong. Harvey’s benchmark defines completion as passing every criterion, so the 19.17% figure is a demanding measure of end-to-end performance, not a claim that the system gets only one-fifth of all details right.
That distinction matters to law firms evaluating tools for research, contract review and drafting. A high criterion pass rate may indicate useful performance on individual elements, but it does not establish that a draft is ready for a client, court or counterparty. The benchmark result cannot replace checking authorities, facts and requested work product before use. It also does not establish how the system will perform on a particular firm’s matters, document sets or workflow.
Price makes the comparison relevant for procurement, but a low benchmark cost does not settle the purchasing decision. Firms also need to understand how client data is handled, which services or external tools receive it, what review controls apply, and what time is required to verify output. Those issues are not answered by the leaderboard score.
Products Worth Considering
Logitech MX Brio Ultra HD 4K Webcam, Graphite
Samsung 27-inch Odyssey G5 QHD Gaming Monitor (G51F)
Amazon Kindle Scribe Colorsoft, 64GB, 11-Inch Color Display with Premium Pen
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Tasks
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. Its tasks pair an instruction with a client matter and ask an agent to produce work for review. Harvey says the public collection contains 1,250 tasks across 24 practice areas, graded against more than 75,000 expert-written, binary pass/fail criteria. Criteria can cover facts, conclusions, citations, deadlines, recommendations, dollar amounts and formatting.
Under Harvey’s all-pass rule, a task is complete only if every required criterion passes. As a result, a single failed requirement can make the whole task unsuccessful. Vals’ public leaderboard uses a private holdout set intended to mirror the public benchmark mix, according to the supplied source material. The 19.17% result is therefore not the proportion of individual rubric items passed; Vals reports that separate measure as 92.5% for GLM Legal, with a criterion counted as passed only when both judges agreed.
Harvey’s initial baseline, reported May 26, recorded a best overall all-pass rate of 7.1% among the frontier models it tested, and Harvey said those models completed fewer than 10% of tasks end to end in aggregate. That was a separate evaluation and run, so it should not be treated as a direct comparison with Vals’ later leaderboard. It does offer context for why a 19.17% score can rank highly while still leaving most benchmark tasks incomplete under the strict rule.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its grading rule
Limits of the Ranking and Evidence
The leaderboard establishes a score under a particular benchmark setup; it does not show how often the model makes legally consequential errors in ordinary use. The supplied material does not include the full task-level results, a breakdown by practice area, or enough detail to assess which types of criteria account for failures. Nor does a high aggregate criteria rate show whether remaining errors are minor or serious.
The source report describes a small, four-task spot check in which the legal model cited rulings the author could not find. It also reports that two Claude models cited a real ruling for the opposite of what it held, while the OpenAI and Claude models in that check invented none. This was not a benchmark or representative sample; it cannot establish the comparative reliability of the systems. The supplied source also says Z.ai’s pages reviewed described outputs as drafts for legal professionals and did not carry a “not legal advice” notice, but that observation does not settle the model’s terms or users’ legal obligations.
It remains unclear what comparison set Z.ai used for its open-source leadership claim, whether weights are available, and how the model’s data handling and tool connections work in specific customer deployments. The documentation page has no date, and the source material does not provide a later leaderboard result or a response from Z.ai to the concerns about citations and safeguards.
What Firms Should Verify Next
The next useful evidence would be more detailed independent results, including performance by task type, the distribution and severity of failures, and updated leaderboard entries using the same protocol. Any new figures should be compared only when the benchmark version, evaluation set, judges and scoring method are clear.
For law firms considering a trial, the immediate next steps are operational: confirm the model’s availability and access terms, determine whether it connects to external legal research or company-data services, and review contractual and data-protection provisions. Firms would also need a process for checking every authority, citation and material factual claim before generated work leaves the organization. The available sources do not state when Z.ai will publish further benchmark results or whether the model’s weights or additional safeguards will be released.
Key Questions
Does 19.17% mean GLM Legal got only one-fifth of its answers right?
No. It means the model passed every required criterion on 19.17% of benchmark tasks. Vals separately reports a 92.5% pass rate for individual criteria, using a stricter agreement rule that requires both judges to pass a criterion.
Why did the model rank seventh if it passed so few tasks completely?
The ranking reflects performance relative to other models on the same leaderboard, not a threshold for professional readiness. The benchmark’s all-or-nothing rule counts a task as incomplete if even one required criterion fails, which can produce low task pass rates across the field.
Is the 12.5-percentage-point improvement confirmed?
The archived leaderboard figures support the calculation: GLM Legal scored 19.17% and the GLM-5.3-Flash base model scored 6.67%, a difference of 12.50 percentage points. This confirms the arithmetic for those listed results, not performance on every legal task or in live practice.
Does the result show that GLM Legal is ready for unsupervised legal work?
No. A benchmark ranking does not establish that output is safe to use without review. The score measures performance on a particular holdout evaluation, and the source material does not establish real-world reliability, legal responsibility, or the suitability of unsupervised use.
What should a law firm check before testing the model?
Firms should review client-data handling, connected services, access terms and verification procedures. They should also test the model on relevant tasks and require qualified review of citations, facts and other consequential claims before using generated work.
Source: ThorstenMeyerAI.com
