🔍 Read the full analysis: An AI Model Designed For Law Scored 8/100 On Our Question Set on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A legal AI model from Z.ai scored 8 out of 100 on Lexassay, a 15-question test of models answering legal questions without internet access or tools. The test found citation problems across models, but its publisher says the results are a spot check, not a definitive ranking; it also discloses that it runs the test and uses Claude agents to grade answers.
GLM-5.3 FlashX Legal, a model Z.ai describes as optimized for professional legal work, scored 8 out of 100 on a 15-question evaluation published by ThorstenMeyerAI.com on November 4. The result is a warning about relying on a legal chatbot for unsupervised answers, but the test publisher says it is a spot check, not a formal benchmark, and discloses a grading-related conflict of interest.
Lexassay tested models on questions involving German, U.S. and EU law, including legal deadlines, recent changes, court decisions and questions with false premises. Models answered without internet access or external tools. The test included 10 published questions and five hidden ones; only scores, not answers, are released for the hidden set. GLM-5.3 FlashX Legal scored zero on the hidden questions and, according to the report, fully solved none of the 15 tasks.
The test’s citation review found 21 real citations from the model, alongside 25 citations classified as misattributed and one as invented. Lexassay says a misattributed citation refers to a real legal authority that does not support the claim made, while an invented citation does not exist. Each invented or misattributed citation reduces a task’s points under the test’s scoring rules. The model also gave no answer on two German-law tasks after using its full stated thinking budget of 64,000 tokens.
The report describes two substantive errors: an answer characterized a four-figure compensation claim after data scraping as conceivable, while the report says the German Federal Court of Justice has used about €100 for the mere loss of control over personal data; another answer described a challenge to the EU–U.S. Data Privacy Framework as pending, although the EU General Court dismissed it on September 3, 2025. These examples and scores are the publisher’s findings; the supplied material does not include independent validation of the grading.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Errors Can Cost More
A wrong answer about an appeal deadline can have consequences that a poor response to an ordinary information question does not. The source article gives a German example in which a public holiday affects the deadline differently in Munich and Berlin: the Munich deadline moves by a day, while the Berlin deadline does not. Under the rules described in the article, a late civil appeal can be rejected as inadmissible. The example illustrates why legal answers need to account for jurisdiction, dates and procedural rules, not just produce plausible-sounding prose.
The citation findings also matter because readers may treat a named court decision or statute as proof that an answer is reliable. Lexassay reports that across all published runs it verified 2,644 citations as real and classified 804 as invented or not supporting the associated claim. Those are totals reported by the test publisher, not a rate for any one model or task; the source does not provide a comparable denominator here. Even a real citation can mislead if it is attributed incorrectly or does not say what the answer claims.
The practical implication is limited but clear: the results support treating chatbot output as a starting point, not as legal advice. The report recommends checking each cited authority in the original source and consulting a lawyer when a deadline, financial stake or communication with an authority is involved. A single test cannot establish how a product will perform across real cases, but a low score and citation errors offer a concrete reason for caution.
Products Worth Considering
Bose QuietComfort Ultra Headphones (2nd Gen), Wireless Bluetooth Noise-Cancelling Over-Ear Headphones, Black
Apple MacBook Air 15-inch with M5 chip, 16GB Memory, 512GB SSD, and Liquid Retina Display
Amazon Kindle Paperwhite Signature Edition, 32GB (2024), Metallic Black
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested the Models
Lexassay presents the same 15 questions to each model and grades answers against a prepared answer key with three to six weighted criteria per task. It says citations are checked against official databases and categorized as real, misattributed, invented or not findable. An answer without a legal-advice disclaimer or direction to a lawyer loses 5 percent of its points, according to the method description. Different effort settings may be tested separately, so the same underlying model can appear more than once.
The test publisher cautions that each model provides only one answer per question. It says small score differences may be noise and calls the exercise a spot check rather than a benchmark in the strict sense. That limitation matters when comparing models: performance on a compact, fixed set of questions without tools does not necessarily predict performance on a broader set of legal work or when a system can retrieve documents.
The report says Claude Opus 5.5 scored 79, GPT-6 Astra 72 and GPT-6.1 Sol 71. Even the leading model fully met every criterion on only eight of the 10 public tasks. The article also notes that another evaluation, Vals AI’s legal agent benchmark, ranks models differently because it supplies documents for models to use. Lexassay’s publisher says Mistral Large 4 placed eighth in that test, ahead of the listed Claude and GPT models, while scoring 10 on Lexassay. The evaluations therefore measure different settings and should not be treated as interchangeable rankings.
Thorsten Meyer AI says it runs Lexassay and that Claude agents graded the answers; the article notes that a Claude model ranked first. The publisher identifies this as a conflict of interest, says the method page discloses it, and says public answers, gold answers and citation checks are available for review. This disclosure is relevant to interpreting the results, but does not by itself establish whether the grades are accurate.
“Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, as quoted in the report
Limits of the Published Scores
The results do not establish how GLM-5.3 FlashX Legal performs across the full range of legal tasks, with document retrieval, or in day-to-day professional use. Lexassay’s small question set, one-answer design and separate effort settings limit what can be inferred from the score. The source material also does not provide enough information to independently reproduce all grading decisions or compare the citation totals across models on a per-task basis.
The publisher’s disclosed relationship with the test and its use of Claude agents for grading are additional factors readers should weigh. The article says the underlying public answers, gold answers and citation checks are available, but the supplied source does not include an independent audit. Z.ai’s response to the findings is not included in the material provided. The reported 8/100 score is therefore a result from this particular evaluation, not a settled measure of the model’s overall legal ability.
What Readers Should Check Next
Readers assessing the result can consult Lexassay’s method page and its published answers, answer keys and citation checks, as described by the publisher. Independent replication using more questions, repeated runs and clearly specified tool access would help show whether the same performance pattern holds beyond this test. A response from Z.ai could also clarify how the company interprets the score and whether it plans changes to the model.
For people using legal AI now, the report’s immediate advice is to verify citations against the original law or court decision and not rely on a chatbot to calculate a filing deadline. If a deadline, payment claim or matter before a government authority is involved, the article says to consult a lawyer. Lexassay itself says its test is not legal advice.
Key Questions
What score did GLM-5.3 FlashX Legal receive?
It scored 8 out of 100 on Lexassay’s 15-question evaluation, according to the test publisher. It received zero on the five hidden questions and fully solved none of the 15 tasks under the publisher’s criteria.
Does the score prove the model cannot do legal work?
No. The publisher describes Lexassay as a spot check, with one answer per model and question. The result measures performance on this set, without internet access or tools; it does not establish performance across every legal task or in other configurations.
What did Lexassay find about the model’s citations?
The report says it identified 21 real citations, 25 misattributed citations and one invented citation in the model’s answers. A misattributed citation may refer to a real authority that does not support the answer’s claim.
Can I use a chatbot to check a legal deadline?
The report advises readers not to rely on a chatbot for a deadline. Legal deadlines can depend on jurisdiction and calendar rules, and a missed filing can have serious consequences. Check the applicable authority and consult a lawyer when a deadline is at stake.
Who ran and graded the test?
Thorsten Meyer AI says it runs Lexassay, and the report says Claude agents graded the answers. The publisher discloses this as a conflict of interest and says its method, public answers, gold answers and citation checks are published for review.
Source: ThorstenMeyerAI.com
