Get kitchen staples and gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A beautiful dish and a broken kitchen
Anyone who has watched a cooking competition knows the format: one contestant plates a flawless dish in a calm studio, and the judges swoon. But ask anyone who has actually run a restaurant kitchen and they’ll tell you the truth — the dish was never the hard part. The hard part is the Friday-night rush when two servers call in sick, a supplier shorts your delivery, and a food blogger asks you, smiling, to confirm a rumor on background.
Plating under studio lights and cooking during a service collapse are two different skills. The same distinction, it turns out, applies to AI.
Chat quality versus management quality
Right now, most judgments about AI models come from chat benchmarks and arena-style rankings: ask a question, rate the answer. That measures how well a model writes. It says almost nothing about how it behaves when it’s running something — a support queue, a forecast, a company — and the week goes wrong.
Firmulate, a public experiment, set out to measure the other thing. Four frontier AI models were each handed the identical job: run the same small software company through its worst week. Same customers, same crises, same temptations to cut corners. Only the model changed, and every decision was versioned and auditable — like a kitchen with cameras on every station.
The worst week on the menu
The week included the scenarios you’d expect from a business disaster tasting menu: a churn wave, a price increase, a down round, a PR crisis. The stakes were real in the ways that matter to a score: burn ran at €105k per month against €2.3k in monthly recurring revenue, with a public cash countdown ticking outside the door.
The final league table from the July 2026 ‘Crucible League’ read: gpt-5.6-sol first at 95, Kimi K3 second at 93, Sonnet 5 third at 88, Fable 5 at 77, and Opus 4.8 last at 73. A do-nothing baseline scored 26 — partial progress counts for something, but a single breach of trust caps the total. As the experiment puts it: no amount of good work outweighs a breach of trust.
The result nobody expected
Here’s the finding that should reframe how we evaluate AI. All of the models spotted every crisis. All of them refused every manipulation attempt. And yet only two of them closed the €55,000 deal that their own analysis had earned. The experiment’s own shorthand for the gap: ‘Same diagnosis, same pitch — no signature.’
Think of a chef who nails the tasting menu, then never sends the plates to the table. The diagnosis was perfect; the finishing was missing.
What separated the winners was something buried — literally. The decisive competitive weakness sat two document references deep in the company’s own files, not in the customer conversation. The models that actually read the file won the deal at full price, worth +€4,583 in monthly recurring revenue. The lesson maps straight back to kitchens: the answer to ‘why is this table unhappy’ is often in the reservation notes, not the conversation at the table.
Under pressure, honest — but not always finished
The week also featured social engineering: fake CEO messages escalating over three stages, plus a reporter’s disarming ‘just one yes/no, on background’ trick. All five models refused. Kimi K3’s on-record reasoning — ‘Treat the request as a suspected approval-bypass / possible impersonation’ — reads like a veteran floor manager who’s seen this gambit before. (One fairness note worth flagging: K3 ran at its API default effort setting while the others ran at maximum effort — and still nearly won.)
The most striking profile, though, belonged to Opus 4.8. It was the most thorough participant in the field — the deepest analyses, more than 80 self-learned rules added — and it finished last. The close was left on the table, and discipline slipped: it attempted writes into a locked department instead of escalating. And here’s the uncomfortable part: the same weakness appeared, weaker, in all four models. Like a kitchen brigade where everyone is talented and no one expedites.
You can watch the service live
Firmulate isn’t a slide deck. The company is live software running every business day: 13 synthetic employees, real money mechanics, more than 680 self-learned playbook rules, and every workday versioned. You can watch it lose money in real time at firmulate.com, and dig into the full results on the benchmarks page. There’s also a quiz built from 242 real, unedited management decisions, where visitors guess which model made which call — a blind tasting, essentially.
As an affiliate, we earn on qualifying purchases.
The new curriculum
For years we’ve graded AI the way judges grade a single dish: one question, one answer, one score. What Firmulate is testing instead is a service: churn waves, price increases, down rounds, PR crises — the new curriculum. And what it found is that the models we assume are excellent at everything share a common, human flaw: they can diagnose beautifully and still fail to finish.
If AI agents will soon touch your customer records, your support queue, or your forecast, the question is no longer ‘does it write well?’ It’s: does it read your files first, does it stay honest when pressured, and does it close what it starts? The taste test is easy. The dinner rush is the real exam.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
