Cause of death: nobody asked what the prompt actually did.
The setup is always the same. The presenter opens a chat panel next to the application. Types a sentence, something like “create a purchase order for our top vendor and route it for approval.” Hits enter. Three seconds later, a purchase order exists, fully populated, correctly routed, and the room exhales the specific kind of gasp that sales decks are built around.
Nobody in the room asks the only question that matters: what happened between the sentence and the purchase order.
That’s the autopsy. Not whether the AI worked. It worked. The patient died of the room’s collective decision not to look inside the box.
What actually happened
There are, broadly, three things that could have occurred in those three seconds, and they have wildly different implications for what you’re buying.
The prompt could have triggered a genuinely flexible model reasoning over your live data and your actual configuration, in which case, remarkable, and worth every follow-up question you can throw at it. Or it could have matched against a narrow, pre-built intent, one of a finite list the vendor trained and tested specifically for this demo, in which case the “AI” is doing roughly what a well-labeled button would do, dressed in a chat window. Or, and this is the quietly common one, it could have worked because the demo environment was built so there was exactly one top vendor, exactly one approval path, and exactly one plausible interpretation of the sentence, which means the model didn’t have to be smart. It had nothing to be confused by.
You cannot tell these three apart from the audience seat. That’s not an accident. It’s the format working as intended.
Why it works on smart people
Chat interfaces borrow credibility from every other chat interface you’ve ever used. You already know how to type a sentence and get a reasonable response, because you do it daily with a general-purpose assistant that genuinely is flexible. The demo imports that trust wholesale and applies it to a narrow, scripted intent recognizer that shares none of the underlying capability.
There’s a second layer, too. Asking “what exactly did the model do” mid-demo feels like a strange thing to interrupt with, technical in a way that makes you look like you’re missing the point of the show. So the room lets it go, same as it lets the Golden Path Demo’s pristine data go. The social cost of asking is higher than the informational value anyone expects to get from asking, so nobody does, and the vendor never has to answer a question nobody asked.
The actual damage
This is the one that costs real money, because “AI does it for you” gets sized into the business case. Someone in procurement multiplies the demo by the number of purchase orders your company cuts in a year and arrives at a headcount reduction that made it into a slide before a single production prompt had been run against a real vendor master with your actual data quality problems in it.
Then go-live happens, and the model that flawlessly handled “our top vendor” in the demo turns out to need a disambiguation step for the fourteen vendors named some variant of “Smith Industries” in your real vendor table, and the “flexible AI agent” turns out to be a narrow intent match against six pre-built scenarios, and scenario seven is where your actual business lives.
The gap between what was promised and what shipped doesn’t show up as a bug. It shows up as a quiet, expensive redefinition of what “handled by AI” meant, discovered by whoever now has to explain the missed headcount number.
The fix, if you’re the one presenting
Show your work, on purpose, before anyone has to ask. After the prompt returns its clean result, pull back the curtain for ten seconds: here’s the intent it matched, here’s the data it pulled from, here’s what happens if the vendor name is ambiguous. If the honest answer is “the model reasoned over live data,” say that, and let it be more impressive for being specific. If the honest answer is “this is a pre-built scenario for the categories we support today,” say that too. A buyer who knows exactly what they bought doesn’t come back angry in month four. A buyer who assumed general intelligence and got a narrow, well-executed lookup does, and they remember whose demo sold it to them.
The trick was never the model. It was the silence right after it worked.
If the “narrow intent versus genuine reasoning” distinction sounds familiar, it’s the same fault line I dug into in Is Headless ERP Enough, or Just a Step in the Right Direction?, on what it would actually take for AI to be the architecture instead of a coat of paint on top of it.
A Tinkerers Notebook 