Questions worth asking before you sign.
These are drawn from the NIST AI Risk Management Framework, which names valid and reliable, accountable and transparent, and explainable among the traits of trustworthy AI. A vendor who cannot answer them is telling you something.
On evidence
- How do you evaluate accuracy, on what data, and how often? Ask for the artifact, not the adjective.
- What does the system do when it is unsure? A number with no uncertainty behaviour is a number that will be wrong silently.
- Show me a case where it got something wrong and what happened next.
A vendor with real evaluation practice will have all three ready, because their own engineers need them.
On data
- Where does our data live, and in whose account?
- Is any of it used to train models, ours or anyone else's? Get this in the contract rather than the call.
- Who at your company can read a customer call, and what is logged when they do?
- What is your deletion process and how long does it take?
- Are you a business associate under HIPAA for any part of this? If the answer is a pause, that is your answer.
On the human gate
- What can this system do without a person approving it? List everything.
- Can an administrator disable the approval step? Who has that permission?
- Is the approval recorded, and can we produce it two years later for an examiner?
In a regulated line the third question is the one that matters most and gets asked least. An approval that leaves no record is an approval you cannot demonstrate.
On the exit
- If we stop tomorrow, what stops working?
- What do we get back, in what format, and how fast?
- Does anything you built live only inside your product?
The exit answer predicts the relationship better than the demo does. A vendor confident in the value they add is relaxed about how you leave.
One question for yourselves
Who on our side owns this after the pilot? A tool with no internal owner degrades quietly, and the vendor gets blamed for it eighteen months later. Name the person before you buy.
Want this looked at on your own floor?
Twenty minutes on your systems and your numbers. You leave with the map either way.