The Demo Always Works: How to Tell a Real AI System From a Rehearsed One
Practical questions to ask an AI vendor that reveal whether their demo reflects a working system or a carefully staged path through one.
Every AI demo works. That is what a demo is for. The vendor picked the documents, the customer record, the question, and the moment to click. None of that is dishonest, but none of it is evidence either. Your job in that meeting is to find out whether there is a system underneath the performance, and you can usually do it in thirty minutes with the right questions.
Bring your own input
The most useful thing you can do in a vendor meeting is take the keyboard. Hand the system a document from your own business: a messy contract, a support thread with three people talking over each other, an invoice from your worst-formatted supplier. Ask them to run it live, unprepared.
Watch what changes. Response time, hedging, formatting, the confidence of the answer. A system that holds its shape on an input nobody staged has met the real world at least once.
If the vendor declines, ask why. "Your data isn't loaded into the sandbox" is a fair answer. Accept it, then put the test in writing as a condition before you sign, with your files and your definition of a pass.
Ask what happens when it is wrong
Ignore the accuracy percentage for a moment and ask about the remainder. If the system is right 92% of the time, what does the other 8% look like from the user's chair? There are only three options: it says it does not know, it escalates to a person, or it produces a confident wrong answer.
Systems that fail loudly are cheap to run. Systems that fail quietly are expensive, because the cost lands on the customer who acted on bad information and the employee who spends a week untangling it.
Then ask to see a case where the system refused to answer. Teams with a working product usually show these with some pride, because restraint is the hard part to build.
Ask where the answer came from
Anything touching your own material — policies, pricing, contracts, product specs — should cite its source. Ask for a citation, then open it. The question is not whether a link appeared. It is whether the cited document actually says what the answer claims.
Do this twice. Two checks tell you more about a system than an hour of slides.
Ask about the day after the data changes
Demos run on a frozen snapshot. Your business is not frozen. These questions separate a product from a prototype:
- How does new or corrected information get in, how often, and whose job is that?
- When a document is deleted or superseded, how long until the system stops using it?
- Does the system respect your existing permissions, or does everyone get the same answers regardless of role?
- When the underlying model changes, do you get to test before it reaches your users?
- Who is on call when it breaks at nine on a Monday, and what is the response time?
Vague answers here are the strongest signal you will get all meeting. A team that has run something in production has lived through all five.
Ask for numbers that cost something to produce
Percentages on a slide are free. Ask for numbers with a shape: how many users, over how many months, measured against what baseline, and who measured it. "Resolves 60% of tier-one tickets" means little until you know what the other 40% cost to reroute, and whether that figure came from a pilot with ten people or a year with a thousand.
Good vendors answer with specifics and caveats. Someone who has shipped a real system knows exactly where it strains, and says so without being pushed.
Ask what it cannot do
Ask the vendor to describe the use cases where their system is a poor fit. A confident team has this answer ready and it is usually precise: long documents in scanned form, languages outside their evaluation set, anything requiring a judgment call with legal exposure.
A team that says it handles anything is telling you they have not yet run it anywhere hard enough to find the edges. You will find them instead, after the contract is signed.
The last question
Before you leave, ask to see what an existing customer sees on an ordinary Tuesday. Not the demo: the dashboard, the logs, the queue of escalated cases, the report someone reads on Friday morning. Real systems accumulate this unglamorous furniture because someone had to operate them. A rehearsed demo has no Tuesday.
We sit on the buyer's side of that table often, and the pattern holds: the systems that survive these questions rarely look impressive in a demo. They look boring, specific, and instrumented. That is what working looks like.