A customer types “resize jpg in Acrobat.” Another asks, “Unable to delete PDF content need help.” Neither resembles the carefully named task topics we tend to write in technical documentation. Both came from real customer queries.
The gap between the language customers use to ask questions about products and the language documentation teams use sits at the center of a 2024 study from researchers at the University of Massachusetts Amherst and Adobe Research.
Their paper, KaPQA: Knowledge-Augmented Product Question-Answering, examined whether adding product-specific knowledge to a retrieval-augmented generation system could improve answers to questions about Adobe Acrobat and Photoshop.
 | Kapqa Knowledge Augmented Product Questi... 1.37MB ∙ PDF file |
|
Customer questions rarely match the way documentation is written. Before the AI can search for an answer, it has to guess what the customer is asking. To help with that guessing, the researchers built a simple knowledge base of relationships pulled from the documentation itself — things like rotation handle → rotates → text box — and used it to rewrite vague customer questions into clearer ones before searching.
It worked, but barely. And in some cases, the "help" made things worse.
One customer in the study wanted to split a large PDF into smaller files. The system pulled in product knowledge about reducing file size — a related but different task — and rewrote the question into “How can I reduce the size of a PDF file?,” sending the search in the wrong direction.
The docs had the answer that the system needed. The rewriting step introduced the confusion, by pulling in information that was topically close to the question but not what the customer actually meant.
Even when the system retrieved the correct documentation, that wasn’t the end of the hallucination story. If the correct source showed up lower in the list of results handed to the AI, it was more likely to generate a wrong answer.
And for questions that were already clear and well-formed, the same “helpful” generative AI rewriting step made answers worse, not better. Often, it nudged good questions in the wrong direction.
