AI Agent Task Success
The share of eligible support tasks an AI agent completes correctly end to end, including the answer, policy decision, tool actions, and resulting system state.
Task success % = correctly completed eligible tasks / eligible tasks attempted × 100; repeated-run reliability = tasks passed in every one of k trials / tasks tested × 100AI agent task success asks whether the customer's job was actually completed, not whether the model produced a fluent reply. The evaluator checks the full end state: the right answer, the right policy application, every required tool action, no prohibited action, and a correct handoff when the task cannot be completed safely. This makes it stricter than containment and more meaningful than a vendor-labelled automated resolution.
Build the denominator from versioned, eligible cases sampled across intent, risk, language, customer state, and edge conditions. Score critical failures separately so a fabricated refund or unauthorized account change cannot average out against easy successes. For nondeterministic systems, repeat the same cases and report reliability across runs; a system that succeeds once but fails on the next attempt is not production-stable.
What it hides: the test set defines reality. Teams can inflate success by excluding difficult intents, using synthetic cases that do not resemble production, or allowing a model to grade its own outputs without human calibration. Pair offline task success with production escalation, repeat contact, correction, complaint, and customer-outcome measures, and publish every eligibility rule and excluded failure mode.