CSAT surveys are a reasonable proxy for human support quality. They measure whether the interaction felt good to the customer, which correlates well with whether the problem was actually resolved when a person is doing the work. The correlation breaks down with AI-handled tickets, and understanding why changes how you should think about measuring your AI's performance.
When a customer receives an AI response, several things can happen that produce a positive CSAT score without genuine resolution. The response arrived quickly. It used confident, polite language. The customer interpreted it as resolution and gave a 5-star rating without verifying whether the answer was correct. They come back with the same problem three days later. That second ticket does not appear in the CSAT data for the first interaction.
Why CSAT specifically misleads for AI responses
Human support agents are calibrated by feedback loops that AI agents are not. When an agent gives a wrong answer, they usually find out about it. The customer responds, escalates, or the problem reappears. Over time, agents develop a sense of when they are sure versus when they are guessing. That uncertainty is usually reflected in their tone and their willingness to say "let me check on that."
AI agents do not automatically communicate uncertainty unless they are explicitly designed to. An agent that gives a confident-sounding wrong answer and then closes the ticket will receive the same CSAT prompt as an agent that gave a correct answer. The customer rates the speed and tone of the response, not the accuracy of the resolution. If your AI is fast, polite, and confident, your CSAT scores for AI-handled tickets may look better than for human-handled tickets even if the resolution quality is lower.
This creates a dangerous measurement illusion. Teams that optimize purely for CSAT can be building a system that deflects a lot of tickets, scores high on satisfaction surveys, and quietly erodes customer trust through repeated partial resolutions. The erosion does not show up until customers start churning or telling colleagues your support is unreliable.
What resolution fidelity measurement looks like in practice
Resolution fidelity is the question: did the customer's problem actually get resolved? For AI-handled tickets, you can approximate this in several ways, none of them perfect but all of them better than CSAT alone.
The most direct signal is ticket recurrence rate. If the same customer or the same account opens another ticket on the same topic within a defined window, say 14 days, the first resolution probably failed. Tracking recurrence rate by topic for AI-handled versus human-handled tickets gives you a quality comparison that is independent of how the customer felt about the interaction.
A second signal is escalation-after-closure. If a customer marks a ticket as resolved or closes it without escalating, but then opens a new ticket referencing the same issue, that is a soft resolution failure. Capturing this requires connecting ticket threads across sessions by account and topic, which your helpdesk may or may not support natively, but it is worth building if your volumes warrant it.
The third signal is source document quality for AI answers. When Quack answers a ticket, it references specific source documents. If those documents are outdated, have low internal link authority, or were last updated more than 90 days ago, the answer they produced deserves lower confidence weighting regardless of how fluent the response sounded. Auditing source document quality by topic cluster gives you a leading indicator of where resolution fidelity is likely to suffer before the CSAT numbers reveal it.
Building a two-layer measurement framework
We recommend treating CSAT as a first-pass signal and resolution fidelity metrics as a second-pass signal. CSAT tells you about the interaction quality. Resolution fidelity tells you whether the underlying problem was addressed. Both matter. Neither alone gives you a complete picture.
For AI-handled tickets specifically, the second layer deserves more weight than it does for human-handled tickets, precisely because AI responses are more likely to produce the fast-and-confident-but-incomplete failure mode. A support operation that scores an 88% CSAT on AI tickets but has a 22% recurrence rate in that cohort has a quality problem that the CSAT score is hiding.
Practically, this means building a weekly or monthly report that shows AI-ticket CSAT alongside AI-ticket recurrence rate and escalation-after-closure rate, segmented by topic cluster. When you see a topic cluster where CSAT is high but recurrence is also high, that is a signal that the AI is giving plausible-sounding answers that do not fully resolve the issue. That is the calibration problem to address, either by updating the source documentation or adjusting the confidence threshold for that topic category.
What Quack surfaces in analytics
Our analytics dashboard is built around this two-layer model. You see deflection rate and CSAT alongside recurrence rate, escalation-after-closure rate, and source document age for each answer category. The goal is to make the second layer as visible as the first so that quality problems surface before they accumulate in customer churn.
We are not saying the current implementation is complete. We are actively working on tighter recurrence detection and better source document freshness signals. But the principle is fixed: CSAT alone is insufficient for measuring AI support quality, and any AI support tool that only reports CSAT is showing you less than half the picture.
If you are running a pilot of any AI support tool, including ours, build the recurrence rate measurement before you go live. It is the most honest indicator of whether your customers' problems are actually getting resolved, and it is the metric that will matter most when someone asks whether the AI is doing good work.
