A Gartner survey of 413 marketing technology leaders published in October 2025 found that 45% say their AI agent vendor tools fail to meet the business performance their vendors promised. The same research found that half of organisations lack the technical and data stack readiness required for AI agent deployment – a gap the demo almost never surfaces.
The AI vendor demo is not designed to reveal limitations. It is designed to demonstrate capabilities under optimal conditions, with curated data, in a controlled environment. The gap between demo performance and production performance is where most AI vendor disappointments originate.
This guide gives you six questions every AI vendor must answer – in writing, not in a sales presentation – before procurement proceeds.
The Six Questions
1. What Does Your Production Performance Look Like on Data Like Ours?
The demo uses the vendor’s own data, or a curated example dataset that presents their system favourably. Ask for documented production performance – accuracy, latency, error rate, uptime – from customers with data characteristics similar to yours. Industry, data volume, data quality, and system architecture all affect AI performance significantly.
A vendor who cannot provide reference performance data from production customers is asking you to fund their proof of concept. That is a legitimate business arrangement, but it should be priced and contracted as a pilot, not as a production deployment.
2. Who Are Your Reference Customers in Our Industry and Size Range?
Request three customer references with direct relevance to your situation – same industry, comparable company size, similar use case. Contact them directly, without the vendor on the call. Ask specifically about implementation timeline vs. projection, ongoing support quality, data quality requirements encountered, and whether they would deploy again.
A vendor who hesitates to provide direct reference contacts, or who provides references from significantly different contexts, is a signal to investigate further before proceeding.
3. What Data Do You Need, and What Happens If Ours Doesn’t Meet the Requirement?
Every AI system has data requirements. Minimum dataset size, specific data fields, acceptable quality thresholds, format requirements. Ask for these in writing before deployment begins. Then assess your own data against them honestly.
The follow-on question is equally important: if your data does not meet the requirement at deployment, what is the contractual position? Does the vendor assist with data preparation? Does the performance guarantee still apply? Is there a defined path to meeting the requirement, or does non-qualifying data result in an underperforming system with no recourse?
4. What Are the Real Integration Dependencies and Timeline?
The demo shows the AI working in isolation or against a clean API. Production deployment connects the AI to your ERP, CRM, authentication system, data warehouse, and potentially several other systems. Each connection has dependencies, access requirements, and the possibility of undiscovered complexity.
Ask the vendor for a technical integration specification document – not a slide deck, a document. Ask them to identify every system they will need to connect to and the requirements for each connection. Ask for a realistic timeline based on comparable integrations they have completed, not their standard sales slide. The gap between the timeline on the sales slide and the timeline in the technical specification is usually significant.
5. What Are Your Data Handling and Security Practices?
As covered in previous content on AI data privacy: ask for the data processing agreement before negotiating price. Confirm subprocessor disclosure – which third-party AI providers will process your data. Confirm data storage location and jurisdiction. Confirm whether your data will be used for model training or improvement. Confirm the deletion process at contract end.
If the vendor does not have a data processing agreement ready, they are not ready for enterprise procurement. This is not a negotiating position – it is a minimum standard for any AI tool that will process business data.
6. What Does the Contract Say About Performance Guarantees and Exit?
The performance guarantee in an AI contract is only as useful as its specificity. ‘Industry-leading accuracy’ is not a guarantee. ‘95% classification accuracy on your production data within 60 days of go-live, with a defined remediation process if not met’ is a guarantee. Ask for specific, measurable commitments with defined remediation terms.
Exit provisions matter equally. What is the data export process at contract end? What is the notice period? Is there a minimum contract term with penalties for early exit? An AI system that captures your data in a proprietary format and requires 12 months notice to exit has substantial hidden lock-in costs that are not visible in the initial pricing conversation.
Red Flags to Watch For
Three vendor behaviours in the evaluation process are reliable indicators of problems in production: reluctance to provide direct reference contacts, inability to produce a data processing agreement, and unwillingness to commit specific performance metrics to the contract. Any one of these warrants a harder look before proceeding. All three together is a clear signal to pause.
Frequently Asked Questions
How long should AI vendor evaluation take?
A thorough vendor evaluation – including reference checks, technical specification review, data processing agreement review, and legal review of the contract – should take 4–8 weeks. Shorter timelines typically mean steps are being skipped. Vendors who pressure for faster decisions deserve more scrutiny, not less.
Should we use a procurement consultant for AI vendor selection?
For significant deployments – above £100,000 total contract value, or any deployment touching regulated data – independent procurement or technology advisory input is worth the cost. The fee for independent advice is typically recovered in the first contract negotiation.
What is the most important thing to check in an AI vendor contract?
The performance guarantee and the data handling terms are equally important. A contract with no specific performance commitment gives you no contractual recourse if the system underperforms. A contract with inadequate data handling terms exposes you to regulatory and reputational risk. Both must be acceptable before signing.
Comments are closed