Auto-QA scores 100% of tickets — should you trust it?

Every auto-QA demo follows the same script. The vendor pipes in a week of your tickets, the dashboard floods with green, and a number lands on screen: 100% of conversations scored, no sampling, no backlog. After years of grading a token handful by hand, that can feel like the future arriving. The question nobody asks in the demo is the one that matters back at your desk — scored against what, and would a human have agreed?

The pitch is coverage, and the pitch is fair

Manual QA has always carried an embarrassing secret: most teams grade a sliver and extrapolate to the whole. On average, just

of interactions get a human review, and from that fraction we hand out scores, rank agents, and decide who needs coaching. A sample that thin isn't a safety net; it's a rumour. Auto-QA's core promise — read every ticket, every agent, every day — is a genuine fix for a real problem. Coverage is not the hype. Coverage is the one thing the machine unambiguously does well.

The trouble starts when coverage gets mistaken for accuracy.

Where it holds, and where it drifts

Auto-QA is most reliable on the categories that are really compliance checks. Did the agent verify identity, follow the refund process, link the right article, close with the correct disposition? These are the deterministic, evidence-is-in-the-transcript items, and a model reads 100% of them without fatigue. That happens to be where scorecards spend most of their attention — "solution / resolution" is the single most commonly evaluated category, scored by

By contrast, empathy shows up on fewer than half of scorecards, and empathy is exactly where automated scoring drifts. A model can spot the phrase "I understand your frustration"; it cannot reliably tell a warm, well-judged apology from a cold one that happens to contain the right words. The further a category sits from process compliance and the closer it moves to tone, judgment, and context, the wider the gap between what the model scores and what a human would.

A number without a rubric is just a fast opinion

Here is the uncomfortable part. A 100% score is meaningless until you know what it measures and whether it agrees with anyone. The industry's own benchmark for review quality sits around

so a tool that hands every agent a shiny 96 has either quietly disagreed with the human baseline or redefined what "good" means. And QA is not a proxy for happiness, either: across

analysts found QA scores did not correlate with CSAT. The two measure different things — process adherence versus customer sentiment — which is fine, as long as you never let an auto-QA number masquerade as either.

Calibrate it like you'd calibrate a new reviewer

Treat the model as a new hire on the QA team, not an oracle. Before you trust its scores, calibrate: have your humans grade the same batch the model graded, category by category, and measure the agreement. High agreement on process items, visible drift on the subjective ones — that is the expected, healthy pattern, and it tells you which categories to keep human. Re-run the check monthly. Models drift as your product, macros, and policies change, and a rubric that fit in January quietly rots by summer.

Then use the coverage for what coverage is good at. Auto-QA's real gift isn't the grade; it's the ability to surface the interesting five tickets out of five thousand — the outliers, the repeat pain, the coaching moments a 2% sample would never have caught. That feeds the part of QA that actually changes behaviour. Only

of teams currently bring review feedback into 1:1 coaching, and a firehose of auto-scored tickets does nothing on its own. A human who uses that firehose to find the right conversation, and sits with an agent to talk it through, does.

Full coverage is worth having. Just don't confuse scoring every ticket with understanding any of them. Pair the machine's reach with a human-anchored rubric and honest calibration — start with why reviewers disagree and the sampling math that auto-QA is meant to replace — and 100% coverage becomes an asset instead of a very confident guess.