It’s a familiar scene. Someone plugs an assistant into a sales team’s entire documentation: product sheets, release notes, tender responses, clinical studies. They ask it a question live. Four seconds later: the answer, with its source. The head of sales smiles. Everyone smiles.
The next day, a rep asks it a question a biomedical engineer asked the day before: is the current software version compatible with the hospital’s PACS? The assistant says yes. Confidently.
Except it’s wrong. A release note from last year says the opposite, and nobody ever added it to the knowledge base.
In a demo, the person showing it off picks the question. They pick a good one. The documents are clean, up to date, and consistent. In the field, it’s the other way around: questions come in badly phrased, documents contradict each other, and the right answer is buried in a PDF nobody filed. And the rep has no time to check, because the buyer is waiting.
So before a team starts relying on a documentation assistant, I recommend a very simple test: the twenty-question test.
Take twenty real customer questions from the last three months of email. Questions your team already knows the answer to, and nothing confidential. Put them to the assistant exactly as written, typos, clumsy wording, and all. Then sort every answer into one of four buckets: right and sourced, right but unsourced, wrong, or “I can’t find it.”
The number that matters isn’t how many it got right. It’s how many wrong answers it gave with confidence. An assistant that says “I can’t find it” is a good colleague. One that makes things up with confidence puts your customer relationships at risk.
The test takes about twenty minutes. It tells you two things: what’s missing from your knowledge base (usually release notes and answers you’ve already sent), and whether the tool knows its own limits.
Which leaves the decision that matters most: who signs off?
My rule at the start is simple: the assistant prepares, the rep sends. Nothing goes to a customer until a human has read it and owns it. It sounds slow. In practice, checking a sourced answer takes seconds, while hunting for the right product sheet often takes several minutes. You keep the time saved, and you keep the accountability.
The template: the twenty-question test
Copy this table into a spreadsheet and run the test with two reps, not on your own.
If more than two of the twenty answers are wrong and stated with confidence, don’t put anything in front of a customer. Fill the gaps in the knowledge base first.
Seen in the field
The questions that trip an assistant up most aren’t the technical ones. They’re the ones that depend on a date: “Is that still true since the latest version?” If your knowledge base doesn’t keep track of dates, the assistant can’t guess them.