Running QA when AI writes half the replies

Every support QA program was designed around one assumption: a human wrote the reply, and you're checking their work. Pull a random sample, score it against a rubric, coach the misses. That model was already thin —

— and when AI starts drafting a real share of your replies, it stops working entirely. A 2% sample of human-authored tickets tells you nothing about the model that touched the other 98%. Here's how to rebuild QA for a queue where the machine writes first and the human decides.

What actually changed

The unit of work is now a collaboration. When an agent sends an AI-drafted reply, three things are being judged at once: the model's draft, the agent's edit, and — most importantly — the agent's decision about whether to trust it. Your old rubric scores one author. You now have three.

AI raises the floor and hides the ceiling. The strongest evidence we have says assistance helps novices most.

That's good news and a QA trap. AI-drafted replies are uniformly competent: polished, on-tone, plausible. Reviewers can no longer skim for obvious badness. The dangerous failure is no longer the sloppy reply — it's the confident, well-written, wrong one. Fluency now hides error instead of signalling competence.

Speed pressure gets worse, not better. AI makes replying faster, which makes the temptation to send-without-reading stronger.

The rebuild, in five moves

1. Score 100%, sample the review. Automated QA can score every conversation, not 2%. Do it — but don't confuse scoring with reviewing. Let the automated pass triage; have humans deep-review a stratified sample: some high-confidence AI resolutions, some escalations, some low-score outliers. You're sampling for review, not for coverage.

2. Add "verification" to the rubric. Write explicit criteria for the new failure mode: Did the agent catch a factual error in the draft? Did they check the knowledge-base claim the model made? Was an AI answer sent that never should have been? Reward the catch, not just the clean send — an agent who rejected a bad draft did the highest-value work in the queue, and your rubric should be able to see it.

3. Score the model separately from the agent. Keep two ledgers. One tracks AI draft quality before human editing — that's your vendor's real report card. One tracks the final sent reply. The gap between them is the value your agents add. If that gap is near zero, either your model is superb or your agents have stopped reading, and only the transcripts tell you which.

4. Calibrate the automated scorer, relentlessly. An automated QA score is itself a model output, and it drifts. Have your best reviewers blind-score the same conversations the machine scored, and measure agreement. If the scorer and your calibrators disagree more than they agree, your "100% coverage" is 100% noise.

5. Close the loop into coaching, fast. The point of scoring every conversation was never the score. It's that coaching can finally be specific and immediate instead of a monthly sample and a vibe. Feed the patterns back — to agents ("you're over-trusting drafts on billing tickets") and to whoever owns the model ("it's inventing refund policy for the EU region").

The number that keeps you honest

Whatever you build, anchor it to an outcome, not a score. A benchmark helps —

— but an internal quality score that climbs while repeat contacts climb alongside it is measuring your rubric, not your quality. Tie QA to whether the resolution held: no reopen, no repeat contact, no quiet churn. That's the only score the customer ever actually gives you.

QA in the AI era isn't lighter work — it's a promotion. You've gone from spot-checking typing to auditing a human-machine system, which is exactly the skill the whole operation is about to need. On which roles grow and which change, see the org chart after AI.