🔍 Read the full analysis: Can We Trust LLMs To Judge Whether A Research Idea Is Novel? on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Ai2 researchers tested six large language models as judges of scientific research novelty and found their results could change sharply with evaluation choices. In one pairwise test, small prompt wording changes shifted accuracy by more than 50 percentage points; literature retrieval and additional reasoning did not consistently improve performance. The team released a benchmark, data, code and a pipeline, while cautioning that some benchmark labels are imperfect.
Ai2 researchers report that six large language models gave unstable judgments of scientific idea novelty when prompts and other evaluation choices changed, raising questions about using automated scores to screen research proposals. In one pairwise experiment, changing a few prompt words shifted accuracy by more than 50 percentage points; the researchers also found that literature retrieval and additional reasoning did not consistently improve judgments.
The study tested GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6 using two approaches: asking models to rate an idea’s novelty on its own and asking them to compare ideas in pairs. The authors say the results depended on choices such as prompt wording and evaluation format. The reported prompt effect comes from one pairwise experiment, not a finding that every prompt change produced the same shift.
For one benchmark, the team used ICLR 2026 peer reviews to identify 154 ideas from accepted papers in the top-rated 10% of their research areas and 145 ideas from rejected papers in the bottom 10%. The selected reviews praised originality in the high-novelty group and raised originality concerns in the lower-novelty group, without comments pointing the other way. An annotator agreed with the direction of the extracted novelty signal in 47 of 50 checked cases; the authors say the three disagreements did not change a review’s classification.
A second setting compared the 154 human-authored ideas with 154 proposals from a basic language-model generator. The researchers treated the generated ideas as less novel, but described those labels as weak and not individually verified. In a separate expert check of 30 pairs, selected heavily from cases the strongest judge got wrong, an expert preferred the human idea in every pair and judged none of the generated ideas novel. The study also reports that two purpose-built novelty systems performed worse than a simple prompted Claude Opus 4.6 baseline.
Why Prompt Stability Matters
Research teams may use an LLM judge to narrow a list of candidate ideas before spending time on literature searches, experiments or expert review. If a score shifts substantially after a small prompt edit, it may reflect the evaluation setup as well as the proposal’s originality. That makes it harder to compare scores across projects or to treat a single result as dependable evidence that an idea is new.
The finding also bears on efforts to automate parts of scientific discovery. Systems that generate hypotheses or synthesize research need ways to check whether their proposals add something beyond existing work. The study does not show that every LLM novelty assessment fails; it shows that strong results on one chosen setup are not, by themselves, evidence of reliable judgment across setups. Researchers and institutions considering these tools have reason to test how sensitive conclusions are to prompts, models and comparison methods.
Products Worth Considering
LG UltraGear 27-Inch QHD OLED Gaming Monitor, 240Hz, 0.03ms, G-SYNC Compatible, FreeSync Premium Pro, HDMI 2.1
Jackery Explorer 300 Portable Power Station with 40W Solar Panel
Amazon Kindle Paperwhite Signature Edition (12th Generation, 2024), 32 GB, Metallic Black
As an affiliate, we earn on qualifying purchases.
How Ai2 Built Its Test Sets
The researchers describe novelty judging as assessing whether an idea is new relative to previous work, either as a standalone proposal or in comparison with another idea. They say there is no standard evaluation recipe: researchers can select different models, prompts, reasoning budgets, formats and literature-retrieval systems. They also argue that evaluation should reflect intended uses, including judging proposals generated by AI, rather than relying only on human-written papers.
The benchmark uses reviewer comments as evidence of relative novelty, not as a definitive measure of originality. The authors caution that acceptance or rejection alone cannot establish whether a paper is novel, since publication decisions may reflect other factors. Ai2 released the paper, benchmark data, code and an automated pipeline intended to help others build further evaluation sets. The study forms part of the institute’s broader work on tools to assist scientific discovery.
““seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.””
— Ai2 study authors
Limits of the Novelty Evidence
The benchmark’s labels are signals from peer reviews, not a gold-standard determination of whether an idea is original. The generated proposals in the second test set were assumed to be less novel than the selected human ideas, and the authors acknowledge that this was not checked individually across the full set. The expert review covered 30 pairs, not all benchmark examples.
The supplied study description does not give full results for every model, prompt, reasoning budget or evaluation format, and it does not state the calendar date the work was released. It also leaves open whether the findings carry over to other disciplines, different idea generators or systems with broader access to research literature. The authors say some conclusions based on disagreements between evaluation setups do not depend on benchmark labels, but the available description does not detail how widely that result held across settings.
Further Tests Across Research Fields
The release gives other researchers materials to test whether the reported instability holds across fields, idea sources and judge designs. Follow-up work could compare prompts and evaluation methods across more kinds of proposals, examine how literature access affects results, and assess whether any design makes judgments more consistent without relying on weak labels.
Ai2’s release description does not announce a specific follow-up study or timeline. For now, the benchmark provides a basis for independent testing, while the study leaves unresolved which combination of model, prompt, retrieval and evaluation format, if any, can support dependable novelty judgments.
Key Questions
What did the Ai2 study find?
Six tested LLMs gave unstable judgments of research-idea novelty across different evaluation setups. In one pairwise experiment, changing a few prompt words shifted accuracy by more than 50 percentage points.
Did literature retrieval or extra reasoning fix the problem?
The authors report that adding literature retrieval or giving models more reasoning did not consistently improve their judgments. The supplied description does not provide full results for every configuration.
How did the researchers label ideas as more or less novel?
One test set used ICLR 2026 peer-review comments praising originality or raising originality concerns, alongside the papers’ ratings and acceptance outcomes. The authors treat those reviews as evidence of relative novelty, not a definitive standard.
Are the benchmark’s AI-generated idea labels verified?
No. The researchers treated proposals from a basic language-model generator as less novel than selected human-authored ideas, but said those labels were weak and not individually verified. An expert assessed a separate sample of 30 pairs.
Does the study prove LLMs cannot judge novelty?
No. The findings point to sensitivity to evaluation choices, but do not establish that every LLM novelty assessment fails or show which approach, if any, can produce dependable judgments across fields and settings.
Primary source: Hugging Face · via ThorstenMeyerAI.com
