Questions to ask your AI support agent vendor about quality

• Ask how the vendor defines “resolved” and “contained,” and what evidence backs each count.
• Ask how it detects wrong answers, loops and dead ends, and how often it reviews them.
• Ask when and how the bot hands off to a human agent, and what context travels with the customer.
• Ask for conversation-level evidence behind every metric, on your own data.
• Plan an independent check of the bot’s conversations, side by side with your human teams.

Use these questions in the evaluation itself and again at renewal. Group them into five areas, and score each answer on whether the vendor shows you evidence or only describes a feature:

1. Definitions. What the vendor counts as success.
2. Failure detection. How it finds wrong answers, loops and misunderstood requests.
3. Handoffs. How customers reach a person, and what happens next.
4. Evidence and transparency. Whether every number traces back to real conversations.
5. Independent checking. Whether someone other than the vendor can grade the work.

These areas track the characteristics NIST lists for trustworthy AI systems, which include “valid and reliable,” “accountable and transparent,” and “explainable and interpretable” ¹. You don’t need to adopt the framework to use them. They simply give your questions a shared vocabulary your security, legal and support teams already recognize.

Start with definitions, because every metric in the vendor’s dashboard depends on them.

1. How do you define a resolved conversation, and who decides? A strong answer names the rule, shows a few conversations that passed and failed it, and lets you change the rule to match your policy. A weak answer says “the customer didn’t come back.”

2. What counts as a contained conversation? If the vendor counts every conversation that never reached a person, that count also includes customers who gave up. Ask how it separates customers who got an answer from customers who closed the window or reopened the issue on another channel.

3. Which metrics do you report, and which of them do you grade yourselves? List every metric in the proposal and mark the ones the vendor’s own model scores. Those are the numbers you’ll want checked independently.

Confident wrong answers deserve their own questions, because they look like success in a containment report.

4. How do you catch answers that are wrong or against our policy? Ask what the bot checks its answers against, how a wrong answer gets flagged, and who reviews the flags. Ask to see five real flagged conversations from a comparable deployment, with customer details removed.

5. How do you find loops, dead ends and misunderstood requests? A loop is the bot repeating itself while the customer rephrases. A dead end is a conversation the bot can’t move forward. Ask how each is detected and reported, and whether you can see the full transcript behind each flag.

Handoff quality decides whether a bot failure becomes a recovered conversation or a lost customer.

6. When does the bot hand off to a person, and can we change those rules? In a Gartner survey of 3,566 B2B and B2C customers conducted in February and March 2026, 87% said it’s essential for companies using GenAI in customer service to provide an option to reach a human agent ². Ask what triggers a handoff, whether customers can ask for a person at any point, and how you adjust the triggers.

7. What context travels with the customer at handoff? The human agent should see the transcript, the detected intent and anything the bot already tried. Ask to watch a handoff end to end, and ask how the vendor reports conversations where the customer had to repeat themselves after reaching a person.

For a deeper walkthrough of this step, see our guide on how to audit AI agent handoffs to human agents.

Every number the vendor reports should open into the conversations behind it.

8. Can every metric be traced to specific conversations? If the dashboard says the bot resolved a share of billing questions, you should be able to open those billing conversations and read them. NIST notes that transparency answers “what happened” in a system and explainability answers “how” a decision was made ¹. Ask for both.

9. Will you run your evaluation on our past conversations? A sample of your real, de-identified conversations shows how the bot handles your products, policies and edge cases. Agree up front on what a pass looks like, and keep the answers you get as your baseline.

Independent checking matters because the vendor that sells the agent has a stake in how its results look.

10. Can we export full transcripts and flags so a third party can grade them? You’ll want the raw conversations along with the summary metrics, so an independent grader or your own QA team can score them on your terms.

11. How will your reporting line up with how we grade our human agents? Your human agents and BPO teams are probably already graded on a scorecard. AI agents fail in different ways, so they need their own rubric, but you’ll want both views side by side so you can compare outcomes across the whole operation. Our guide to what AI agent QA is covers what that bot rubric should include.

A bot that passed evaluation can drift as your products, policies and knowledge base change.

12. What happens to quality when we change a policy, a product or a knowledge article? Ask how the vendor tests changes, how it tells you when quality drops, and how you’ll find the affected conversations. NIST describes reliability as correct operation “over a given period of time, including the entire lifetime of the system” ¹, which is why a launch-day evaluation isn’t enough on its own.

Use this table to score each vendor’s answers during the evaluation. A strong answer shows evidence on your data; a red flag is a feature description with nothing to open.

Here’s how three hypothetical vendors might answer “How do you define a resolved conversation?”

• Vendor A: “A conversation is resolved when the customer doesn’t contact us again within 72 hours.” This is a proxy. It counts customers who gave up and moved to email as resolved. Score it low.
• Vendor B: “Our model decides whether the issue was resolved, and we report the rate weekly.” This is a vendor-graded metric with no visible rule. Ask for the rule and the conversations behind it before you score it.
• Vendor C: “Here’s the written rule, here are ten conversations that passed and ten that failed, and here’s where you can edit the rule to match your refund policy.” This answer gives you evidence and control. Score it high, then confirm it on your own conversations.

Not every deployment needs the full twelve questions. A lighter review fits when:

• The bot only answers simple, low-risk questions, such as store hours or order status, and hands everything else to a person.
• Volume is low enough that your QA team can read a meaningful share of bot conversations by hand each week.
• You’re running a short pilot with a clear end date and a plan to evaluate properly before you expand.

The full set fits teams putting an AI agent in front of a large share of customers, handling regulated topics such as identity, payments or privacy, or running AI agents alongside outsourced teams where several parties report on their own work.

unitQ Support QA is an independent grader for support conversations. It grades human agents on your scorecard and AI agents on their own bot rubric, side by side in one product, independent of the vendor that built the bot ³.

• An independent bot audit. It audits the AI-agent conversations that appear in the support data you connect, independent of the bot vendor, and checks whether containment was real and flags wrong information, loops and dead ends, inappropriate escalations, misunderstood issues and inaccurate resolutions, plus how often conversations recover after reaching a person ³. Bot Audit in unitQ Support QA is configured for your workspace during setup.
• Your rubric, in plain language. Your team defines each check, which answers count as a fail, and worked examples, and owns and edits the rubric in Tag Studio in unitQ Support QA.
• Evidence behind every score. Every scored miss links back to the conversation behind it.
• Coverage without the sampling. It scores the conversations you connect, not a sample, across written support conversations such as chat, email and messaging.
• Connects to your stack. unitQ integrates with support platforms such as Zendesk, Salesforce Service Cloud, Intercom and Freshdesk ⁴.

On data protection, unitQ’s security page states: “unitQ is SOC 2 Type II compliant” and “unitQ is compliant with the EU General Data Protection Regulation (EU GDPR)” ⁵.