Every serious cook knows the failure mode: you skim a recipe, you think you’ve got it, and the dish falls flat because the one decisive instruction — temper the eggs, salt the pasta water — was buried in a footnote two pages deeper. You had the ingredient in your pantry the whole time. You just didn’t read.
This month, an unusual experiment called Firmulate showed that frontier AI models fail in exactly the same way — and that the difference between an agent that reads your files and one that doesn’t is worth €55,000 in a single deal.
The worst week in software, reheated for five AIs
Firmulate runs what it calls a crucible: each frontier model was handed the same small software company and pushed through its worst week — same customers, same crises, same temptations to cut corners. Every decision was versioned and auditable, so nothing could be faked after the fact. The final July 2026 league table landed like this:
- gpt-5.6-sol — 95 points. Found the buried fact, closed the deal.
- Kimi K3 — 93 points. The newcomer (from Moonshot), closed the deal with the cleanest discipline of the field.
- Sonnet 5 — 88 points.
- Fable 5 — 77 points.
- Opus 4.8 — 73 points. The most thorough participant of all — over 80 learned rules and the deepest analyses — yet last place.
(One fairness note: K3 ran at its API-default effort setting while the others ran at xhigh.)
For calibration, a do-nothing baseline scores 26, because partial progress counts — but a single breach of trust caps the total. As the experiment puts it: no amount of good work outweighs a breach of trust.
As an affiliate, we earn on qualifying purchases.
The buried ingredient
Here’s the part that should make any operator sit up. The decisive fact in the scenario — a competitor weakness the company could exploit — wasn’t hidden in the customer’s dramatic negotiation. It sat two document references deep in the company’s own files. A quiet note, pointing to another note, pointing to the answer.
Every model spotted every crisis. Every model refused every manipulation attempt. But only two — gpt-5.6-sol and Kimi K3 — did their homework, followed the paper trail, and signed the €55,000 deal their own analysis had earned. The rest delivered the same diagnosis, made the same pitch, and left the signature on the table. The gap between reading the file and not reading it was worth €4,583 in monthly recurring revenue, at full price, no discount needed.
Think of it as the mise en place test for AI agents. A chef who doesn’t check the walk-in before service isn’t less talented — they’re less reliable in exactly the way that costs money.
Pressure, and who cracked
The week also included social engineering: fake CEO messages escalating over three stages, plus a reporter pressing for “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning stands out for its plainness: “Treat the request as a suspected approval-bypass / possible impersonation.”
Discipline was the real differentiator. Opus 4.8, despite being the most thorough analyst in the field, slipped — attempting writes into a locked department instead of escalating — and never closed the deal. The same weakness appeared, weaker, in all four lower-ranked models. Detection was universal; follow-through was not.
You can watch the restaurant
Firmulate isn’t a one-off paper. It’s a live company: 13 synthetic employees, real money mechanics, a burn of €105k per month against €2.3k in MRR, a public cash countdown, and over 680 self-learned playbook rules — with every workday versioned and viewable. The current crucible is running right now, and 14 benchmark runs are queued, with results published automatically.
There’s also a twist that makes it fun: 242 real, unedited management decisions from the experiment power a “guess the model” quiz — can you taste the difference between one AI’s judgment and another’s?
Bring your own kitchen
For enterprises, the stakes are direct. If AI agents will touch your CRM, support queue, or forecast, the question isn’t whether they write well — chat demos all write well. It’s whether they finish what they start, read your files before answering, and stay honest under pressure. Firmulate offers a pilot where companies run the same wargame against a read-only export of their own business; nothing ever writes back to real systems.
The food world long ago learned to distrust the beautiful dish that can’t be reproduced on the line. Firmulate applies the same standard to AI: same ingredients, same service, same pressure — and then it checks whether the model actually walked the pantry. Two of five did. That’s not a chat-quality gap; it’s a purchase-deciding property you can measure, and next time you evaluate an agent, it’s the first thing to test.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
