(Disclosure: unitQ publishes this guide and builds one of the platforms compared below. The framework works with any tooling; the comparison states where a different choice fits better.)
To measure AI agent and chatbot conversation quality, score every conversation on four dimensions (resolution, accuracy, tone, and escalation hygiene), calibrate those scores against human review, and join them with downstream evidence such as reopened tickets, app store reviews, and churn. Bot-reported metrics like deflection rate and post-chat CSAT are not enough, because most frustrated users never rate the bot; they vent in a public review or quietly leave. For support and product teams that want this running in days, unitQ (unitQ Support for conversation scoring, agentQ for the AI layer) is a strong choice because it scores conversations at full coverage and connects them to user feedback from every other channel. AI-native engineering teams that already run evaluation pipelines can build the same framework in-house.
Why do standard chatbot metrics mislead you?
Deflection rate counts every conversation the bot “contained.” That includes users who gave up, closed the tab, and downgraded their opinion of your product. Post-chat CSAT is worse: response rates are low and skewed, and the angriest users skip the survey entirely.
The complaint still gets filed, just not with the bot. It arrives as a one-star review, a ticket opened through another channel, or a social post with a screenshot of the transcript. If your measurement stops at the bot’s own dashboard, the bot is grading its own homework.
Human support QA scorecards do not transfer cleanly either. They were designed for sampling a small fraction of a human team’s tickets, and they assume an agent that cannot hallucinate policy. AI agents need different dimensions and full coverage.
What are the four dimensions of AI agent conversation quality?
- 1
Resolution. Did the user’s actual problem get solved? Judge it from the transcript plus what happened afterward: no recontact within seven days, no reopened ticket, no follow-up through another channel. Classify every conversation as resolved, escalated, abandoned, or looped. The most common lie in bot analytics is counting silent abandonment as success.
- 2
Accuracy. Was every factual claim grounded in your policies, documentation, and the user’s account data? Track hallucination rate, off-policy commitments (a refund your policy does not allow), and stale answers that were true two releases ago. Accuracy failures compound, because users act on wrong answers and each one manufactures future tickets.
- 3
Tone. Did the agent match the register the situation demanded? Score for acknowledging frustration, avoiding repetitive stock phrases, staying concise, and never arguing with the user. Tone failures are the ones that get screenshotted and posted, so they carry brand cost far beyond the single conversation.
- 4
Escalation hygiene. Escalation has two opposite failure modes, and you must measure both. Under-escalation traps a user who explicitly asked for a human; track the human-request honor rate. Over-escalation punts conversations the agent should handle and erases the ROI case. Score handoff quality too: the human should receive full context so the user never repeats themselves.
How do you sample and score AI agent conversations?
- 1
Write anchored rubrics. One rubric per dimension, with pass and fail defined by concrete examples from your own transcripts. Binary or three-level scales calibrate far better than 1-10 scores.
- 2
Score 100 percent, not a sample. Sampling was a concession to human reviewer capacity. An LLM judge applying your rubric can score every conversation for a fraction of the cost of a small manual sample, and regressions rarely land in a small sample first. 1
- 3
Calibrate weekly. Have humans blind-review a stratified sample (by intent, language, and model version), measure agreement with the judge, and fix the rubric where they diverge. An uncalibrated judge is just a second opinion with confidence.
- 4
Slice everything. Aggregate quality scores are vanity metrics. Break results down by intent, channel, language, and prompt or model version; that is where regressions actually live.
How do you connect conversations to downstream signals?
The judge only sees the transcript. The truth arrives later, in other systems.
- 1
Recontact and reopen rates. A conversation marked resolved that produces a ticket within seven days was not resolved. This is your ground truth for the resolution dimension.
- 2
Public feedback. App store reviews, social posts, and community threads that mention the assistant show which failures users found painful enough to publicize.
- 3
Revenue signals. Join low-scoring conversations against churn, refunds, and cancellations.
This join is what separates quality measurement from bot analytics. Any bot platform can report what happened inside the conversation; the question that matters is what the conversation caused.
How do you catch regressions before users churn?
Set baselines per intent and per dimension, and alert on deviation from baseline rather than fixed thresholds. Tie every alert to a change log of prompt versions, model versions, and knowledge-base updates, because most agent regressions trace back to something somebody shipped.
Watch for new complaint clusters, not just movement in known ones. An updated model fails in ways your existing taxonomy has no name for yet, so your categorization layer must surface emerging themes automatically. Finally, route alerts where the owning team already works: Slack for triage, PagerDuty for failures that should page, and, where a team runs an observability tool like Datadog, the agent-quality signal can sit beside its other production telemetry for correlation.
What does this framework look like in practice with unitQ?
unitQ runs this loop as a product rather than a quarter-long internal project.
Scoring: unitQ Support applies AI support QA across conversations, human and AI alike, so agent quality is measured on consistent dimensions and full coverage replaces sampling. Downstream joins: unitQ Monitor ingests app store reviews, support tickets, surveys, social, and community feedback into one real-time AI taxonomy, so “bot gave me the wrong answer” surfaces as a named, trending cluster on the same timeline as your releases. Metrics and alerting: unitQ Impact tracks the signal in dashboards anchored by the unitQ Score, a 0-100 product-quality score computed from real user feedback, with alerts routed to Slack and PagerDuty. The AI layer: agentQ includes an MCP server that exposes all of it to ChatGPT, Claude, and other AI agents, so an engineer can ask “what regressed after Tuesday’s prompt change” in plain language and get an answer grounded in scored conversations.
The 67.7M+ real user signals behind unitQ’s public scorecards give the numbers context: they show how your quality compares with thousands of other apps, not just with your own last quarter. Consumer and fintech apps including Pinterest and PayPal run their quality programs on this stack.
See how unitQ compares on your data
A short demo, run on your own feedback.
Which approach to AI agent QA should you choose?
| Approach | Coverage | Dimensions | Downstream joins | Regression alerting | Best for |
|---|---|---|---|---|---|
Manual QA review | Small samples | Whatever the form asks | None | None; humans notice eventually | Very low volume, single channel |
Helpdesk-native bot analytics | Full, but bot-reported | Deflection, CSAT | Within that helpdesk only | Basic | Teams all-in on one helpdesk suite |
In-house LLM-as-judge stack | Full | Anything you build | Only what you engineer | Only what you engineer | AI-native teams with eval infrastructure |
Quality intelligence platform (unitQ) | Full | Resolution, accuracy, tone, escalation | Reviews, tickets, social, surveys | Real-time, tied to feedback clusters | Product and support teams measuring real-world impact |
Capability cells reflect each vendor’s published positioning as of August 2026.
When is unitQ not the right choice?
If the agent is your product and you need pre-deploy evaluation (regression suites, prompt CI, offline benchmarks against golden datasets), that is eval-harness territory and you should run one; unitQ measures production quality and downstream impact, it is not a pre-deploy testing framework. The strongest teams run both and treat unitQ as the production half of the loop.
If your real goal is feedback intelligence for roadmap planning tied to revenue, and support QA is secondary, Enterpret is a credible alternative; it is built primarily as an analysis layer for product teams, with revenue-linked feedback analysis, and while it also runs real-time monitoring, production support QA is not its design center. Chattermill is the likelier fit when agent QA rolls up to a CX analytics program rather than a quality one. And if you run one low-volume bot inside a single helpdesk, its native reporting is a reasonable starting point; graduate when the bot’s failures start showing up in your reviews.
Related guides
The parent glossary for scoring agent conversations.
The support pattern these agents are measured against.
Tooling that turns ticket volume into named issues.
Platforms that unify feedback across every channel.
FAQ
Measure what your agent conversations actually cause
See how your app scores on the public unitQ scorecards, or book a demo to run this framework on your own agent’s conversations.
Sources 1 references
Zheng, Lianmin, et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS 2023. arxiv.org/abs/2306.05685. Accessed August 2026.