How to Evaluate an AI Automation Partner

Most AI automation pitches look identical from the outside: a slick demo, a chatbot that answers a few pre-planned questions well, and a promise that this will save your team dozens of hours a week. The difference between a vendor who can actually deliver that and one who built a demo for the sales call is almost never visible in the demo itself. It shows up in the questions they ask you, the architecture they describe, and what they say happens when things go wrong.

You don't need to be a developer to run a good evaluation. You need to know which questions separate real engineering from a well-rehearsed script. This is the checklist we'd want a prospective client to run on us.

Is the AI Grounded in Your Data, or Just Prompted and Hoped

The single biggest quality gap in AI automation is between systems that retrieve real information before answering and systems that ask a language model to answer from memory with a clever prompt on top. The second kind looks fine in a demo with five curated questions. It falls apart in production, where it will confidently state wrong prices, wrong policies, or wrong order statuses because the model is pattern-matching, not looking anything up.

Ask the vendor directly: when this system answers a question, what actual data source is it querying at that moment? A credible answer names a specific mechanism — a retrieval step against your knowledge base, a live API call to your order system, a database query — and can describe what happens when that lookup returns nothing. A vague answer ("it's trained on your data" or "it knows your business") is a warning sign. Being trained on something and grounding an answer in a live lookup are not the same thing, and the difference is exactly where hallucinations come from.

This is also where you should ask the question most vendors avoid: what does the system do when it doesn't know the answer? A system with no defined behavior for "I don't know" will guess, and a confident guess is worse than no answer at all, especially in front of a customer. Escalating instead of fabricating is a design decision, not a limitation, and any competent vendor should be able to describe theirs. If you're comparing this for internal use cases like support or research, it's worth reading how a knowledge assistant should actually behave before you sit through more demos.

Questions to Ask About Data Handling and Security

Once an AI system is wired into your business, it usually has access to more than the demo showed: customer records, order history, internal documents, sometimes payment or PII data. Ask where that data lives, who can access it, and whether it's used to train shared models across the vendor's other clients — the answer should be no. If you want a structured, vendor-neutral checklist for this instead of relying on a gut read, NIST's AI Risk Management Framework (see: https://www.nist.gov/itl/ai-risk-management-framework) is a useful reference for the governance and security questions to work through.

Other questions worth asking directly:

• Which third-party AI providers are in the pipeline, and what's their data retention policy for API calls?
• Is data encrypted in transit and at rest, and is there a written incident response process if something leaks?
• Who owns the data, the prompts, and the workflow logic once the engagement ends — you, or the vendor?
• Can you get an export of everything that was built, or does it only run inside their platform?

That last point matters more than it sounds. A lot of "AI automation" is delivered inside a proprietary no-code platform you don't own and can't take with you. That's a legitimate business model, but you should know going in whether you're buying an asset or renting access to one.

Will the "No-Code Automation" Hold Up at Real Volume

No-code and low-code tools are genuinely useful for prototyping and for workflows that run a handful of times a day. The failure mode isn't the tool — it's vendors who build your production process in a tool meant for prototypes and never mention the ceiling you're going to hit.

Ask what happens at 10x current volume. Concretely: what's the rate limit on the platform, what happens when an upstream API times out mid-workflow, is there retry logic, and is there alerting when a run fails silently instead of just not running. A workflow that "usually works" and has no error handling isn't automation, it's a manual process with extra steps and no visibility into when it breaks.

Also ask what monitoring exists once it's live. A workflow that fails at 2am on a holiday weekend and nobody notices until a customer complains is a common, preventable failure. If the vendor's answer to "how will we know if this breaks" is "you'll find out," that's a real gap. This is the same reasoning that should drive how you evaluate any business process automation proposal — the interesting engineering isn't the happy path, it's everything that happens when the happy path doesn't hold.

Red Flags Worth Walking Away From

A few patterns show up repeatedly in engagements that go badly, and they're usually visible before you sign anything:

• Vague scoping. A fixed price and timeline after a single 30-minute call, without questions about your data sources or edge cases, means either a rigid template or a budget that's about to blow through "included" discovery.
• No discussion of failure modes. If error handling, retries, and fallback behavior never come up unprompted, they haven't built anything that's actually run in production.
• Guaranteed results. "Guaranteed to save 20 hours a week" before anyone has looked at your actual volume is a sales number, not an engineering estimate. The FTC has specifically warned businesses about exaggerated AI performance claims (see: https://www.ftc.gov/business-guidance/blog/2023/02/keep-your-ai-claims-check) — a vendor's marketing promise isn't a substitute for your own math. Real estimates come with ranges tied to your specific data.
• No exit path. If you can't get a straight answer about what you own and how to leave, assume you can't.
• The demo only works on the happy path. Ask to see it handle a question it can't answer or a system it's integrated with going down. How it degrades tells you more than how it performs when everything works.

What a Legitimate Discovery and Scoping Process Looks Like

Good discovery is unglamorous and specific. It involves someone from the vendor asking to see your actual data — your documentation, your ticket history, your current process — not just hearing you describe it. It results in a scope that names specific integrations and edge cases you flagged, not generic language that could apply to any business.

It should also produce an honest answer to "should we even build this." A vendor whose discovery process only ever concludes "yes, and here's the contract" isn't doing discovery, they're doing sales with extra steps. Part of a real scoping conversation is figuring out where AI is the wrong tool — where deterministic logic is more reliable, or where volume doesn't justify the build cost yet. If you want to see what that conversation looks like before committing, our process page walks through how we scope engagements, and you're welcome to get in touch and compare it to what you've been pitched elsewhere.

Total Cost of Ownership: Build Cost Isn't the Whole Story

The number in the proposal is the build cost. It is rarely the total cost. AI automation systems need ongoing attention: model behavior drifts as providers update their models, source data changes and needs to be re-indexed, integrations break when a third-party API changes its schema, and usage patterns shift in ways that change what "working well" means six months in.

Ask directly what ongoing maintenance costs, who's responsible for it, and what's included versus billed separately. A fixed monthly retainer with defined scope is easier to budget against than "we'll bill hourly as issues come up." Also ask how model or API cost scales with your usage — a system that's cheap at your current volume can get expensive fast if pricing is per-query and usage grows the way the vendor promised it would.

Build cost buys you a system; ongoing cost keeps it accurate, secure, and running. A vendor who only wants to talk about the first number is setting up a conversation you'll be having again in a year, about why the thing that worked in the demo stopped working in production.

The Underlying Question

Most of these checks reduce to one question: has this vendor actually run something like this in production, under real failure conditions, or have they built something that performs well for exactly as long as a demo lasts? Specificity in their answers about grounding, failure handling, and cost is hard to fake. If you're mid-evaluation and want a second opinion on a proposal you've received, our FAQ covers the questions we get most often, and we're glad to look at what's on the table.