Building Quack required us to understand not just how support documentation is written, but how support questions are asked. Those are two very different languages. Documentation is written by people who know the product thoroughly. Support questions are written by people who are confused, frustrated, or blocked, and who describe their problem in terms of what they were trying to do, not what technical category their issue falls into.
Before we built the first version of our training pipeline, we read 500 support tickets from B2B SaaS products, looking for patterns. We were not trying to build a dataset at that point. We were trying to understand the gap between documentation language and customer language well enough to bridge it. This is what we found.
Customers describe outcomes, not features
The single most consistent pattern in the tickets we read was that customers describe what they wanted to happen, not what feature they were using when it did not happen. A ticket about the webhook configuration might read "our Slack notifications stopped working after we changed our subscription plan." The word "webhook" appears in zero tickets of this type. The customer does not know or care that the underlying mechanism is a webhook.
This matters enormously for support tooling. If your AI is keyword-matching on feature names from your documentation, it will miss the bulk of tickets because the customer used entirely different vocabulary. The AI needs to reason from the outcome description backward to the likely technical cause, which requires understanding the relationships between your product's features and the outcomes customers care about.
We built Quack's retrieval system to match on outcome descriptions as well as feature names, which required building a vocabulary mapping layer during initial setup. That layer is tedious to build but it dramatically improves the AI's ability to retrieve the right documentation for outcome-phrased questions.
The "suddenly" pattern and what it actually means
A large share of support tickets contain the word "suddenly" or equivalent ("out of nowhere," "for no reason," "just started"). This word is almost always inaccurate. Something changed, but the customer does not know what. The change could be a product update you shipped, a configuration they accidentally modified, a browser update, a permission change in their organization, or an expired session.
For documentation writers, "suddenly" is invisible. You cannot write a help article for every possible trigger of every possible unexpected behavior. But for an AI support agent, "suddenly" is a cue to gather more context. The right response to a "suddenly X stopped working" ticket is almost never a direct answer. It is a diagnostic question about what changed in the environment before the problem appeared.
This is one of the hardest patterns to handle with a static FAQ bot. The bot would return a generic troubleshooting article. The AI needs to respond to the specific context of the customer's setup and guide them through narrowing down the cause. Getting this right requires the AI to have internalized not just documentation but the common causal pathways for this class of problem in your product.
Multi-question tickets and the second question problem
A meaningful share of the tickets we read contained more than one question. The second question was almost always implicit. A customer might ask "how do I export my data?" with an unstated follow-up of "and what format will it be in, and will my API keys still work after I do it." The explicit question is answerable. The implicit questions are the ones that determine whether the customer will be back with a follow-up ticket.
Great support agents are good at anticipating the second question. When they explain how to export data, they proactively address the format question and the API key continuity question, because they have seen enough tickets to know those follow-ups are coming. That anticipation comes from pattern recognition built from resolved ticket history, not from documentation.
This is part of why we put closed ticket history at the center of how Quack learns. Documentation tells the AI the canonical answer to the first question. Ticket history tells the AI what questions reliably follow the first one, and what a complete response looks like to a customer who is actually standing at that step in the process.
Tickets that are not questions at all
A meaningful fraction of tickets are not questions. They are statements: "I cannot access my data," "your integration is broken," "I need this fixed today." There is no question mark. There is no diagnostic detail. There is just a description of a broken state and, sometimes, an indication of urgency.
These are the hardest tickets to handle with AI, because the path to resolution requires gathering information the customer did not provide. The AI needs to ask the right clarifying questions in sequence, based on what the stated problem most commonly traces back to. Get the first clarifying question wrong and the customer responds with information that still does not help. Get it right and you can often reach resolution in one exchange.
We use closed ticket resolution threads heavily for this class of ticket. The thread shows exactly what sequence of questions produced a resolution for customers who presented the same initial statement. That sequencing is not in any documentation. It is in the tickets.
Language drift over a product's lifecycle
One pattern we did not anticipate was how much customer vocabulary changes as a product evolves. A feature that launched with one name may acquire a different informal name among users over time. A workflow that was conceptually simple at launch may become more complex as the product adds capabilities, and customers start describing it with new language that reflects the more complex version.
If your AI's knowledge base is primarily based on documentation written at launch and updated only for the most critical changes, there is likely a growing gap between how documentation describes things and how customers currently talk about them. Ticket history is self-updating in a way documentation is not: every new ticket reflects current customer vocabulary, not vocabulary from 18 months ago.
This is a longer-horizon consideration that does not show up in a 30-day pilot evaluation. But teams that have been running their products for more than two years and are now considering AI support should think carefully about when their documentation was last refreshed, not just whether it is technically accurate. Accurate but outdated vocabulary is a hidden source of retrieval failures.
What this changes about how we think about onboarding
When we onboard a new team to Quack, we spend time on ticket vocabulary before we spend time on documentation structure. We look at the most common words that appear in their recent incoming tickets and compare them to the vocabulary in their help center. The gap between those two word sets is a direct predictor of early retrieval quality. A large gap means the AI will struggle at first and requires more closed ticket history to calibrate. A small gap means the documentation is unusually well-aligned with how customers actually talk, and the AI will perform better out of the gate.
We are not claiming you can engineer documentation to perfectly match customer vocabulary before you have tickets. You cannot. But once you have tickets, you can use the vocabulary gap as a diagnostic tool to prioritize which documentation sections to update first, rather than updating the entire knowledge base at random and hoping the AI gets better.
