Most AI automation pitches look impressive in a controlled demo. A chatbot answers five prepared questions, a workflow moves one perfect record from one system to another, and a dashboard shows a clean outcome. The hard part begins after the presentation, when the system meets incomplete data, changed APIs, ambiguous requests, permissions, rate limits, and real users.
You do not need to be a software engineer to evaluate a partner well. You need to ask questions that reveal whether the vendor is designing a production system or packaging a prototype.
Start with the workflow, not the model
A credible partner asks how the work happens today. Who owns each step? Which tools are involved? Where is information copied by hand? What exceptions occur? Which decisions require judgment? What happens when a request cannot be completed?
If the first recommendation is a particular model, agent framework, or no-code platform before those questions are answered, the proposed solution is being shaped around a tool rather than your operation.
Good discovery may conclude that a deterministic workflow is safer than AI, that an existing product only needs better configuration, or that the volume does not justify a custom build. A partner who always arrives at the same solution is not really doing discovery.
Ask how answers are grounded
For knowledge assistants and support systems, ask what source the system checks at the moment it answers. The response should identify a real mechanism: document retrieval, a database query, or a live API lookup.
Phrases like "trained on your business" are not enough. Training and live grounding are different. A model can sound familiar with a topic and still state an outdated price, invent a policy, or miss a changed order status.
Then ask the more important question: what happens when the lookup returns nothing? A safe system should say it does not know, ask a clarifying question, or hand the conversation to a person. Guessing confidently is not graceful degradation.
Review data handling before the demo becomes a deployment
Once connected to a business, an automation may touch customer records, internal documents, order history, or personally identifiable information. Get clear answers to these questions:
- Which systems and data fields are in scope?
- Where is data stored and for how long?
- Which vendors process it?
- Who can access logs, transcripts, and source documents?
- Is client data used to train a shared model?
- How are credentials stored and rotated?
- Can specific sources or records be excluded?
- What can be exported if the relationship ends?
The correct controls depend on the workflow. The important sign is that the partner treats data flow, access, and retention as architecture decisions—not paperwork added after launch.
Make the vendor explain failure modes
Ask to walk through what happens when an upstream API times out, a webhook arrives twice, a record is missing a required field, a user lacks permission, or a model cannot classify a request confidently.
Production automation needs retry rules, idempotency where duplicate work would be harmful, exception queues, alerts, and an owner who knows what to do next. Without those pieces, automation can hide failure more effectively than the manual process it replaced.
Volume matters too. A workflow that runs ten times a day may fit a managed automation platform. One that handles thousands of events, complex branching, or strict latency requirements may need a different architecture. The partner should be able to explain the threshold and tradeoff.
Confirm human escalation is designed, not improvised
AI systems should not attempt every request. Legal judgment, clinical decisions, payment disputes, emotionally charged support cases, and other sensitive work need explicit boundaries.
Ask what triggers a handoff, what context the human receives, and whether the person can override the system. An escalation that drops the user into a blank queue without the conversation or source data simply moves the friction somewhere else.
Clarify ownership and the exit path
Some vendors deliver systems inside a proprietary platform. That can be reasonable, but you should know whether you are buying an asset or renting access.
The agreement should state who owns the code, prompts, data model, integration logic, credentials, and documentation. It should also state how the system can be exported or transferred, and what stops working if the vendor relationship ends.
Scale Hardened builds with ownership in mind: clear documentation, direct access to the people building the system, and an explicit maintenance plan. A client should not discover after launch that the most important workflow in the business cannot leave a black box.
Look beyond the initial build price
The proposal is not the total cost of ownership. Models change, APIs deprecate fields, source documents evolve, users discover new edge cases, and usage volume grows.
Ask who monitors the system, how incidents are handled, what ongoing support includes, and how model or platform costs scale. A lower build price can become expensive if every small change requires emergency work or if usage costs were never modeled.
Red flags worth taking seriously
- Guaranteed savings or ROI before your volume is measured
- A fixed scope after one short call with no access to the workflow
- No discussion of retries, monitoring, or escalation
- Vague answers about where data is stored
- A demo that cannot show an unknown or failure case
- No documentation or handoff plan
- No clear answer about what you own
- Pressure to automate a judgment-heavy decision
The central question is simple: has this partner designed for the moment the happy path ends? Specific answers about grounding, access, failure handling, and ownership are difficult to fake. That is where production engineering becomes visible.
If you are evaluating a proposal, explore how Scale Hardened scopes production systems or book a strategy call to discuss the workflow before choosing the tool.