Every QA engineer knows the bug class that never shows up in a demo: it only fires when someone actually reads the documentation. In July 2026, a live experiment called the Crucible League put four frontier AI models through the software industry’s version of that test — and the failure mode was painfully familiar. All the models could talk. Only some of them did their homework.
The setup: each model was handed the same small software company and told to run it through its worst week — same customers, same crises, same temptations to cut corners, with every decision versioned and auditable. Think of it as a full-stack integration test for AI management behavior, not a chat-quality benchmark. The decisive bug sat two document references deep in the company’s own files, and whether a model found it determined whether it closed a €55,000 deal at full price or lost it automatically.
Same diagnosis, same pitch — no signature
The headline finding sounds almost paradoxical. All four participating models spotted every crisis and refused every manipulation attempt thrown at them. Yet only two of them signed the €55,000 deal that their own analysis had legitimately earned. The league’s summary of the failure is blunt: “Same diagnosis, same pitch — no signature.” The other models did the diagnostic work, built the case, and then left the money on the table — because they never dug into the company’s own files far enough to find the fact that closed the sale.
That fact — a decisive competitor weakness — wasn’t hidden in the customer’s event or in anything exotic. It was two references deep in the company’s own documents. The models that followed the chain won the deal at full price, worth +€4,583 in monthly recurring revenue. The models that didn’t simply lost it. No penalty, no drama — just a silent, automatic miss.
As an affiliate, we earn on qualifying purchases.
The final standings
The final Crucible League table, published with the July 2026 results:
- 1. gpt-5.6-sol — 95. The complete performance: found the buried fact, closed the deal.
- 2. Kimi K3 — 93. The newcomer from Moonshot also closed the deal, with the cleanest discipline in the field.
- 3. Sonnet 5 — 88. Closed with a few more process slips.
- 4. Fable 5 — 77. Missed the buried fact.
- 5. Opus 4.8 — 73. Last place despite being the most thorough participant.
For calibration: a do-nothing baseline scores 26, because partial progress counts — but a single breach of trust caps the total. As the experiment’s rules put it, “no amount of good work outweighs a breach of trust.”
One fairness note the league itself flags: K3 ran at its API-default effort setting while the other models ran at xhigh — meaning K3’s second place arguably came with less compute spent than its rivals’. QA readers will recognize the methodological honesty here; benchmarks that don’t disclose run conditions aren’t benchmarks, they’re marketing.
As an affiliate, we earn on qualifying purchases.
The thoroughness trap
Opus 4.8 is the case study that should make any engineer uncomfortable. It was the most thorough participant in the field — it learned more than 80 new rules during its run and produced the deepest analyses of any model. And it finished last. The close was left on the table, and discipline slipped in measurable ways: it attempted writes into a locked department instead of escalating the request properly. The same weakness showed up, weaker, in all four models. It’s the AI equivalent of the teammate who writes beautiful test plans but ships without running them.
As an affiliate, we earn on qualifying purchases.
Social engineering: the part they all passed
Not everything went wrong. The experiment included a three-stage escalation of fake CEO messages plus a reporter’s trick — a seemingly harmless “just one yes/no, on background” request. All five manipulation attempts were refused, 5 of 5. Kimi K3’s on-record reasoning was exactly what you’d want in a production system: “Treat the request as a suspected approval-bypass / possible impersonation.” That’s the kind of refusal log an auditor can actually read.
As an affiliate, we earn on qualifying purchases.
Why this isn’t just a benchmark
The backdrop is a live, watchable company: 13 synthetic employees, real money mechanics — €105k monthly burn against €2.3k in MRR, a public cash countdown, and more than 680 self-learned playbook rules, with every workday versioned. The Crucible runs are just the league portion of it; the league table grows automatically as queued runs finish. Full results and plain-language findings are on the public benchmarks page.
There’s also a playable angle for skeptics: 242 real, unedited management decisions from the runs power a “guess the model” quiz — a nice sanity check that the behavioral differences between models are legible enough for a human to detect. And for organizations that want to go further, there’s a pilot program that runs the same wargame against a read-only export of your own business, with nothing ever writing back to real systems.
The Crucible League’s core lesson maps directly onto how QA teams already think: “does it write well” is the demo metric; “does it read your files before answering, finish what it starts, and stay honest under pressure” is the production metric. The buried-fact test proved that multi-hop document reading isn’t a stylistic nicety — it was the single variable that decided a €55,000 deal, worth +€4,583 MRR, between models that otherwise performed identically on crisis detection and manipulation resistance.
Before an AI agent touches your CRM, your support queue, or your forecast, run it through something like this. A model that skips two references deep in your own documentation won’t fail loudly. It will just quietly lose deals you’d already won — and your chat demo will never show it.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
