How big does my test set need to be?

Three hundred to five hundred verbatims gives workable per-category signal for your major categories. If a rare category really matters, oversample it deliberately and score it separately; twenty examples of a critical theme beats zero.

What is a good precision or recall score?

There is no universal bar. Match the threshold to the cost of the error: alerting and executive dashboards need high precision so people keep trusting them, while safety and compliance categories need high recall because misses are incidents. Set targets per category before you see any vendor’s results, not after.

Should I trust a vendor’s published benchmark against competitors?

Treat it as an argument, not a measurement. The vendor chose the data, the taxonomy, and the scoring rules. It can tell you what a vendor believes it is good at, which is useful; your own test set tells you whether that holds on your data.

Is human labeling really necessary? It sounds slow.

It is one to two days of work for a few hundred verbatims, and it is the only ground truth you will ever have. It also pays twice: the disagreements between your own labelers reveal ambiguities in your taxonomy that would have poisoned any tool you bought.

What if a vendor refuses to run my test set?

That is a data point. Reasonable vendors may negotiate scope or timing, since a POC takes real effort on their side. But a flat refusal to be measured on your data, on a purchase this dependent on classification quality, tells you how the production relationship will go.

How to evaluate AI accuracy claims from feedback vendors

A vendor-neutral playbook for testing AI accuracy claims on your own feedback. Full disclosure: unitQ sells feedback analysis; the method here is built to be run against any vendor, unitQ included, and we publish it because we’d rather be measured than marketed. ¹

The only accuracy number that matters is the one measured on your feedback, against your taxonomy, by your reviewers. Vendor accuracy claims are usually computed on curated test sets with different base rates, coarser categories, and cleaner text than your production data, so they rarely transfer. The reliable method is simple and cheap: label a few hundred of your own verbatims before any demo, have every vendor classify that same set, and score precision and recall per category yourself. This guide walks through how, and the demo traps to watch for along the way.

Why vendor accuracy numbers rarely transfer

Three quiet variables make any headline number nearly meaningless across contexts.

1
Base rates. If ninety percent of your tickets are password resets, a model that tags everything “login” scores well while missing everything you actually needed to find. Accuracy as a single number rewards guessing the majority class.
2
Taxonomy granularity. Sorting feedback into eight broad buckets is a different problem than sorting it into three hundred specific ones. A vendor measured on the first should not be assumed capable of the second. Ask what taxonomy the claim was scored against, and how many labels it had.
3
Test-set curation. Vendors evaluate on data they chose. That is not deceit, it is selection: clean English sentences, unambiguous complaints, one topic per verbatim. Your reality includes two-word app reviews, sarcasm, code-switching between languages, and tickets that raise four issues in one paragraph.

None of this means vendors are lying. It means their number answers a different question than yours. So ask your own.

The playbook

Step 1Build your test set before the first demo

Pull 300 to 500 verbatims from your real channels in rough proportion to actual volume: support tickets, app-store reviews, in-app feedback, social mentions, survey comments. Do not cherry-pick interesting ones; sample randomly within each channel. Include the ugly stuff, since the ugly stuff is the job. Set the file aside and give every vendor the identical set.

Step 2Decide what “correct” means, in writing

Before anyone labels anything, settle the rules. Which taxonomy are you scoring against, yours or the vendor’s auto-generated one? At what level, top-level category or leaf? Can one verbatim carry multiple labels? Is “billing complaint” tagged on a ticket that mentions billing in passing a hit or a miss? Ambiguity here silently inflates everyone’s scores, because graders resolve doubt generously. Write a one-page labeling guide, then have two people label the set independently and reconcile disagreements. If your own humans agree less than roughly eight or nine times out of ten, tighten the guide before blaming any model; no AI can hit a target your team cannot define.

Step 3Score precision and recall separately, per category

Skip the single blended accuracy score. For each category that matters to you, compute two numbers. Precision: of the verbatims the model tagged with this category, how many actually belong there. Recall: of the verbatims that truly belong, how many did the model find. ² They fail differently and they cost you differently. Low precision floods dashboards with noise until people stop trusting them. Low recall silently hides problems, which is worse, because a missed fraud-complaint cluster is a business incident, not a data-quality footnote. Pay special attention to recall on your rare-but-critical categories: safety, fraud, compliance, data loss. These are exactly where thin training signal makes models weakest and where a miss hurts most.

Step 4Measure coverage alongside accuracy

A model can post strong precision by refusing to classify anything difficult. Check what fraction of your set ended up categorized at all, versus dumped into “other” or left untagged. Fifty percent uncategorized with high precision on the rest is not a good result; it means half your feedback is invisible. Accuracy and coverage trade off against each other, so insist on seeing both.

Step 5Run the live test inside the POC, not the demo

Demos run on the vendor’s data, tuned over months. Insist on a proof of concept where your test set flows through the actual product, configured the way you would run it, ideally connected to one real channel. Then rescore. The gap between demo performance and POC performance is itself a finding: it tells you how much invisible tuning the polished result required, and whether your team can reproduce it.

Step 6Probe the edges deliberately

Salt your test set with known hard cases and track them separately: non-English and mixed-language verbatims, one-line reviews with no context, sarcasm (“great, another update that breaks login”), multi-issue tickets, and domain jargon specific to your product. Also ask how the system behaves when your product changes. A new feature ships, a new failure mode appears, and yesterday’s taxonomy has no bucket for it. Does the system surface emerging themes it cannot yet name, or does the new issue quietly drain into “other” for a quarter?

Step 7Plan the re-test, because accuracy decays

Whatever number you measure at purchase is a snapshot. Feedback drifts, products change, models get updated under you. Keep your labeled set, add fresh examples quarterly, and re-run it. Ongoing audit workflows, human spot-checks of model output on a schedule, matter more over a three-year contract than two points of precision on day one. Ask each vendor how customers audit and correct classifications in production, and who does the correcting.

Claims you’ll hear, and how to test them

The claim	What it usually means	How to verify it
“95%+ accuracy”	A blended score on a curated set with unstated base rates and granularity	Rescore on your labeled set; demand per-category precision and recall
“Human-level tagging”	Comparable to one annotator on easy examples	Compare model output to your reconciled double-labeled set, including hard cases
“Our AI learns your taxonomy automatically”	Auto-clustering that still needs human naming and pruning	Ask to see the raw auto-taxonomy on your data before any analyst touches it
“No training or setup required”	Generic categories that fit every product loosely and yours imperfectly	Check recall on your product-specific and rare-but-critical categories
“Works in 50+ languages”	Strong in a few high-resource languages, thinner elsewhere	Include your real language mix in the test set and score languages separately
“Real-time analysis”	Varies from seconds to daily batches depending on vendor	Push a live verbatim through during the POC and time it yourself

Patterns current as of August 2026; verify every claim against your own labeled test set.

Where we stand

unitQ sells feedback analysis, so read this guide with that in mind. The methodology above is vendor-neutral, and we publish it because we win under it often enough: unitQ’s classification runs in production at enterprise scale for companies like Pinterest, Block, Fidelity, and PayPal, and its granular AI taxonomy is built for exactly the per-category scrutiny this playbook applies. ¹ We will also fail somewhere in your test, because every model does; the honest question is where, how often, how visibly, and how fast the system lets you correct it. Any vendor who discourages you from running your own labeled evaluation is answering that question for you.

See how unitQ compares on your data

A short demo, run on your own feedback.

What is feedback coverage?

The “coverage vs accuracy” trade-off, defined.

What is a feedback taxonomy?

The category set you score a vendor against.

Best customer feedback analysis tools

The field this evaluation applies to.

How to choose a feedback analysis tool

The buying process this evaluation feeds into.

How to build a customer feedback taxonomy

Defining the target before you test any model.

FAQ

See how unitQ scores on your feedback, not ours

Bring a labeled set of your own verbatims and run the evaluation with us, or look up any app’s free public unitQ scorecard.

Run the evaluation with us See your Quality Score

Sources 2 references

unitQ, “Granular AI taxonomy and production classification for consumer-scale customers including Pinterest, Block, Fidelity, and PayPal.” unitq.com. Accessed August 2026.
Google, “Classification: Accuracy, recall, precision, and related metrics,” Machine Learning Crash Course. developers.google.com/machine-learning. Accessed August 2026.