A Vendor-Evaluation Checklist for AI Partners

Almost everything you can see during an AI sales cycle is the part that does not matter, and almost everything that determines your return stays invisible until you are three months in and locked into a contract. The demo is rehearsed. The case studies are survivorship-filtered. The team in the room is the A-team, and some of them will never touch your account again.

So a useful evaluation checklist is not a feature comparison. It is a set of probes designed to surface what a vendor would rather you discover after you have signed. The whole game is to drag a month-four surprise forward into a pre-signature question. The standard procurement checklist cannot do this, because it evolved for static products that are finished the day they are installed. An AI agent is not finished at install: it makes decisions every hour against data that drifts, on a foundation model that gets swapped under it, in front of customers who notice when it is wrong. So the six sections below test the moving target, not the snapshot, and they are ordered around where AI projects actually fail, not where they look impressive.

1. Data, before you let them say the word 'agent'

The fastest way to read a vendor is to watch how quickly they pivot to the agent versus the data underneath it. A vendor who leads with the agent and treats your data as something to 'clean up later' is selling you a demo. An agent grounded on contradictory, un-unified data is a confident liar, and the confidence is exactly what makes it dangerous in front of a customer.

• Ask them to walk through how they establish what is true about a single customer when the answer lives in four systems that disagree. No method for identity resolution and unification means no foundation.
• Ask what they do when your data is not ready. The honest answer is 'we sequence the data work first and it is in scope,' not 'the agent will handle it.'
• Ask who owns data quality after go-live. Data decays continuously; if nobody is watching it, the agent's accuracy rots with it and nobody notices until a customer does.
• Make them show you an engagement where the data work was the hard part. If every story is about the model, they have not run enough of these in production.

This is not a Salesforce-specific point, though it is why a layer like Data Cloud exists at all; it holds on any stack. The capability that separates a real partner from a slick one is whether they treat unified data as the product's foundation or as a footnote. We run the data work first on purpose, and call that phase Agent Ready, because an agent launched on bad data just fails faster and more expensively.

2. The accountability test

This is the single most clarifying question in the evaluation, and most checklists skip it: what number are you accountable for, and what happens to you if it does not move? Watch the answer. A vendor selling effort answers in deliverables: 'a deployed agent, documentation, and training.' A vendor selling outcomes answers in metrics: 'we own lead response time and the conversion that follows, and you will see it weekly whether you ask or not.' Those are different businesses wearing similar logos. Then push one level deeper, because accountability that costs the vendor nothing is theater: is any part of their fee tied to the result?

“

If a vendor cannot name the number they are on the hook for, they are not accountable for your outcome. They are accountable for shipping, and shipping is not the same as working.

An outcome-tied fee is not a billing gimmick; it changes behavior on day one. When your fee depends on a number, you instrument everything from the start, and you refuse to skip the data work to hit a demo date because you are the one who will be explaining the bad number later. That is the model we run at SkySync. The point is self-interest correctly aligned, not virtue, and you should weigh any vendor's incentives the same cold way.

3. The 'day after go-live' team check

Ask one question and listen hard: what does your team on my account look like the day after go-live, compared to the day before? At many firms the senior people you met in the sales cycle roll off at handoff, and you inherit a ticket queue staffed by people who never built the thing. For static software, some of that drop-off is fine; the build is done. For an agent, the day after go-live is when the real work starts, because that is the first time the system meets real customers and you find out every way the first version is wrong. You want continuity from the people who understand why the agent was built the way it was. Get the staffing curve in writing, named roles and hours, before you sign. 'You will have a dedicated team' is a brochure sentence, not an answer.

4. Probe for decay, not just launch

Three forces erode an AI agent after launch, and a serious vendor has a concrete answer for each. An unserious one has never thought past the launch event, because that is when their contract was designed to end.

• Data drift: field usage changes, new products launch, a team starts logging things differently, and the grounding the agent relied on goes stale. Ask how they detect it and how fast they correct it.
• Behavioral drift: the agent starts resolving edge cases in ways nobody intended. Ask how they monitor what it is actually doing in production, and what their process is when it goes off-pattern.
• Platform and model change: the foundation model gets upgraded, the platform ships new capabilities, and last quarter's sensible design now leaves results on the table. Ask who decides whether to adopt those changes, and whether that work is included or a fresh statement of work every time.

The tell is whether 'monitoring and tuning' is a line item with hours and an owner, or a vague reassurance. If the contract structurally ends at launch, you are buying a build and renting decay. We treat that work as its own phase, Agent Care, for exactly this reason: the agent that wins quarter one is not the agent you want running untouched in quarter four. A concrete answer here, with detection thresholds and a named owner, tells you more than any rehearsed product walkthrough.

5. Governance, escalation, and the failure path

Every AI agent will be wrong sometimes. The mature question is not 'how do we make it never fail' but 'what happens when it does, and how contained is the blast radius.' A vendor who claims their agent does not make mistakes is either naive or selling, and both disqualify.

• Guardrails: what can the agent absolutely not do, how is that enforced at the platform level rather than by prompt, and can you change those boundaries without filing a request?
• Escalation: when the agent is uncertain or out of scope, does it hand off cleanly to a human with full context, and how is that path tested before launch rather than discovered in production?
• Auditability: can you reconstruct why the agent made a specific decision after the fact? When a customer disputes an outcome, 'the AI decided' is not a defensible answer to them or to a regulator.
• Kill switch: how fast can you pull the agent, and who on your side has the authority to do it without waiting on the vendor?

Several of these quietly test one thing: whether power stays with you or migrates to the vendor. A partner hands you visibility and control because they are confident the agent earns its place. A vendor who needs you dependent keeps the levers on their side and calls it 'managed.'

6. References, read against the grain

References are theater when you ask the questions the vendor expects. Skip 'were you happy.' Everyone they hand you was happy enough to take the call. They become useful when you ask what the reference was not prepped for: what broke after launch and how the vendor responded; whether the number they were promised actually moved and whether they can still see it today, or whether the dashboard quietly stopped being maintained; and whether you can speak to a client whose project did not go to plan. A vendor who can produce that client and talk plainly about what they learned is more trustworthy than an unbroken wall of wins. And be honest that early-stage and specialist firms carry shorter reference lists, which is not automatically a red flag: the proof that matters is whether the people doing your work have shipped your specific pattern in production, not how many logos sit on the website. Depth on your problem beats breadth across problems you do not have.

The one-page version

If you take nothing else into the room, take these six. Each is chosen because the answer is hard to fake and predicts what you cannot see during the sales cycle. Score vendors on the specificity of the answers, not the polish of the demo. The good ones welcome these questions, because the questions are precisely how they win against firms that only know how to demo; the rest will try to steer you back to the deck, and that alone is most of your answer:

• What number are you accountable for, and is your fee tied to it?
• How do you handle my data before you build the agent?
• What does your team on my account look like the day after go-live?
• How do you detect and fix drift in data, behavior, and the model over time?
• When the agent fails, what stops it, who sees it, and how fast can I pull it?
• Can I talk to a client whose project did not go smoothly?