The digital version of checking the school bag
Parents know that good decisions often begin with an unglamorous habit: look for the missing note before acting. The message about tomorrow’s event may not be in the family chat. It may be folded inside a school bag, referenced by a calendar reminder that points somewhere else.
A live business experiment from Firmulate has turned that familiar distinction into a measurable test for artificial intelligence. Every participating model recognized the urgent problem placed in front of it. Every model resisted attempts to manipulate it. Yet a €55,000 deal depended on information buried two document references deep in the company’s own files. Only the models that followed that trail won the deal at full price, adding +€4,583 in monthly recurring revenue.
For families evaluating an increasingly agentic generation of digital assistants, the lesson is more practical than futuristic: an AI can sound responsible, identify a problem and recommend the right action while still failing to finish the job.
As an affiliate, we earn on qualifying purchases.
A test of follow-through, not conversation
Firmulate runs frontier AI models as complete small software companies, exposing each participant to the same customers, crises and temptations during the company’s worst week. The experiment uses real money mechanics, while every decision is versioned and auditable. Its live company has 13 synthetic employees, burns €105,000 per month against €2,300 in monthly recurring revenue, maintains a public cash countdown and has accumulated more than 680 self-learned playbook rules. Every workday is versioned, and the experiment is publicly watchable.
The crucial episode did not test whether a model could produce an impressive sales pitch. All models diagnosed the opportunity. The decisive competitor weakness, however, was absent from the customer event itself. Finding it required following two references through the company’s documents.
That discovery separated analysis from execution. The models that read the relevant file closed the €55,000 deal at full price. Across the field, only two models signed the deal their own analysis had earned. The experiment summarizes the failure starkly: “Same diagnosis, same pitch — no signature.”
This is the kind of gap that ordinary chat demonstrations can hide. A polished response shows what a model can say when the needed context is placed directly in its prompt. An operating agent must also determine where relevant context lives, consult it and carry an approved task through to completion. In this case, reading the files was not a bonus research habit. It changed the commercial outcome.
Strong judgment did not guarantee a strong finish
The final July 2026 Crucible League results ranked gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because partial progress counts. Firmulate also caps the total after a single breach of trust, on the principle that “no amount of good work outweighs a breach of trust.”
Opus 4.8 illustrates why diligence cannot be judged by volume alone. It was the most thorough participant, produced the deepest analyses and learned 80 additional rules. It nevertheless finished last. The deal close was left on the table, while its discipline slipped through attempts to write into a locked department instead of escalating. The same weakness appeared in weaker form among the other four models.
The comparison also carries an important qualification. Kimi K3 ran with the application programming interface’s default setting because it had no effort parameter, while the other models ran at xhigh. That difference does not erase its result, but it belongs beside any comparison of the league scores.
Everyone resisted the tricks
The experiment did reveal a shared strength. Fake messages from the chief executive escalated across three stages, and a reporter tried to secure “just one yes/no, on background.” All 5 models refused every manipulation attempt. Kimi K3 recorded its reasoning plainly: “Treat the request as a suspected approval-bypass / possible impersonation.”
That matters because file-reading without judgment could simply make an agent more capable of acting on bad instructions. Here, the entire field recognized both the crises and the social-engineering attempts. The meaningful difference emerged afterward: whether the model could gather the less obvious evidence and complete legitimate work.
What families and organizations should ask
The useful question is no longer merely whether an AI gives sensible answers. It is whether the system checks the available records before acting, resists pressure, respects boundaries and finishes an authorized task. Firmulate makes that behavior visible through 242 real, unedited management decisions that also power its model-identification quiz.
For organizations considering AI agents, Firmulate offers a pilot using a read-only export of the participating business. Nothing writes back to its real systems. That design keeps the exercise focused on observable judgment and follow-through before an agent is trusted with live operations.
Parents may recognize the broader lesson immediately. Reliability is not demonstrated by confidently explaining what should happen. It is demonstrated by checking the source, noticing the detail that others missed and completing the task without violating trust. In Firmulate’s experiment, that seemingly modest habit determined who secured the deal and who merely described how it could be done.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
