AI Didn't Create Documentation Debt — It Exposed It

A client described a problem to me in a recent meeting. Their company built AI-powered incident response software for use by cybersecurity teams. Here’s how it works (at a high level): during a live security breach, the tool reads the company’s product information (user guides, specs, and repos) and uses that information to walk an engineer through what to do to address the breach in real time, instead of leaving them to dig through a wiki while the network is on fire.

According to the client, during a string of security incidents this year the tool started giving incorrect instructions that made things worse. For example, the tool told the team to isolate a compromised server using an API parameter that had been retired months earlier. It also walked a customer through re-enabling an encryption setting that the engineers had phased out for compliance last year. And, in another case, it pointed a customer toward what looked like a standard access shortcut — but was not. Instead, the customer was instructed to utilize a leftover authentication bypass built for a sales demo and never removed from the system.

What The Investigation Found

The client told me their engineering team’s first theory was the oh-so-obvious one: the model was hallucinating.

The engineering crew was wrong. After a little detective work, the team discovered that the theory didn’t hold up once someone checked what the AI had retrieved against what was true. To be clear, the information in the hallucination-laden AI generated text was true once-upon-a-time. But, when things changed with the product, that documentation was never updated. Tracing it back to the source turned up technical documentation files that had been sitting there, unmaintained, for quite a while.

Upon closer inspection, they found:

• the retired API parameter still listed in a publicly available integration guide nobody had updated

• the reasoning behind a compliance change that was only once discussed in a Slack thread between two engineers who’d never migrated it to the wiki the AI used as source content — and who both no longer worked at the firm

• the authentication bypass sitting in the codebase with no comment marking it temporary

• the architecture diagrams the model relied on pointing to repositories that had long-since been deleted

On its own, that’s a documentation debt story.

How The AI Reasoned From What It Found

All of the information accessed by the AI was, at least partially, accurate. The facts were fine. The absence of explicit cues and the model’s use of the source content that led it to generate inaccurate answers.

That’s an inference failure, not a retrieval failure.

Handed the exact same content, a human cybersecurity tech writer likely would’ve stopped to pause and ask perhaps questions of the underlying information. Experienced folks whose day-to-day work provides them with knowledge those with lesser experience lack, likely notice when something looked off, is out-of-date, or otherwise, potentially wrong. It’s background work most of us do constantly without paying much attention to it.

The AI retrieved the same clues as the human cybersecurity tech writer and did not stop to pause and ask questions of the content. It moved straight from “this exists in a document” to “this is the instruction,” with nothing in between.

The Role Nobody Replaced

The client told me something else about that team. Before any of this, they’d had a reviewer everyone relied on — let’s call her Betsy, since her real name isn’t mine to share.

“Everyone relied on her because she was so good at it,” the client told me.

Betsy was, in their words, always the person who’d say, “That doesn’t look right.”

She retired recently. Nobody replaced her with someone who had the same eye for detail. Her responsibilities got split across the rest of the team instead, so proofing and review responsibilities became something everyone was expected to own a piece of — rather than something one dedicated person owned.

She wasn’t simply catching typos. Betsy’s value came from context, product knowledge, and years of experience layered on top of ordinary editing — the kind of background that lets someone recognize when something is off even when nothing is technically wrong. The client’s regard for her was so high that I started thinking of her as the queen of that function. Nobody worried about coverage while she held that role, because she was there.