Nothing crashed. When we walk people through this case, that's what they find hardest to accept. There was no outage, no error page, no red on any dashboard. The assistant answered every question in under two seconds, just like on demo day, and it was wrong more and more often.
Think of this post as an autopsy. The patient is a composite, drawn from the kind of internal knowledge assistant we see over and over: a bank's frontline staff ask it questions, and it answers from product terms, fee schedules and compliance circulars. Names and numbers are changed, but the decay is real and the sequence is nearly always the same. What interests me most is that everyone was looking straight at it while it died, and nobody saw a thing.
On demo day it was superb, and you know the scene. Clean, curated corpus. A stakeholder types "what's the early repayment fee on the flexi mortgage," and the assistant pulls the exact clause, cites it and reads it back. The room nods. Someone says "this would have saved my team an hour a day." Sign off happens that afternoon.
A demo is a lie by construction, even when nobody in the room lied. It runs on a frozen snapshot of hand picked documents and friendly, well formed questions. Production runs on a corpus that changes under you, questions typed by a stressed agent with a customer on hold, and a long tail of edge cases nobody scripted. The demo measures your best possible day, and production is every other day.
That gap explains most of why MIT's NANDA initiative found that around 95% of enterprise GenAI pilots deliver no measurable P&L impact. The report blames an organizational and data "learning gap," not a gap in model quality, and says so bluntly. The model didn't get worse between the pilot and the flop. The pipeline around it did, and nobody was watching the pipeline.
The first week in production looked fine, better than fine even. Adoption was climbing. Latency sat where it always did, and the error rate was a flat line at basically zero. In the Monday standup the team pointed at the observability dashboard and moved on to the next project.
This is the trap, so let me be precise about it. Every metric on that dashboard measures whether the plumbing works. Did the query return, how fast, did anything throw? Not one of them measures whether the documents that came back were the right ones. Retrieval quality has no natural alarm. A stale vector queries exactly as fast as a fresh one, so latency stays flat while relevance falls apart without anyone noticing.
Your dashboards are smoke detectors. Retrieval decay is carbon monoxide: odorless, silent, and invisible to a sensor built for smoke.
Everyone had installed smoke detectors and felt responsible for having done so. Whatever was filling the room didn't make smoke.
By week three, frontline staff were routing around it. There was no bug report, just a shrug: "It's a bit hit and miss now, I just check the intranet." Support tickets for the assistant didn't spike, because nobody files a ticket against a tool they've stopped trusting. The usage graph dipped a few percent, which is easy to miss and easy to blame on the novelty wearing off.
When we finally instrumented retrieval properly, the shape was textbook. Recall had drifted from around 0.92 at launch to about 0.74. Documents that ranked second at go live now surfaced at rank eight, below the cutoff of what the model ever sees. A detailed write up of this exact freshness failure puts numbers on it that match what we see: embeddings built on a January corpus lose 15 to 20% of their retrieval accuracy against June queries, purely because the world moved on underneath them.
A vector embedding has no sense of time. Semantic similarity captures what a document is about, not whether it's current. In February the compliance team had issued a revised fee schedule. The old one was never deleted. Worse, it had a three week head start in the index, matched more of the historical queries, and had built up relevance the way a senior employee builds up seniority. So when an agent asked about the early repayment fee, the assistant confidently retrieved and cited the superseded schedule: fast, grounded, sourced and wrong.
Teams almost never plan for the fact that different documents rot at different speeds. An API reference or a fee table can be stale within two weeks, while an architecture doc or a foundational policy stays good for a year or two. With one reindex cadence for the whole corpus, you either pay to re-embed stable content constantly or let your volatile content go off. Usually both.
Before the real cause was clear, the team reached for the fashionable fixes. Most of the wasted money goes here, so the dead ends belong in the autopsy too.
"We need a bigger context window. Just paste the whole policy set in." Tempting, and wrong. Chroma's 2025 Context Rot study ran 18 frontier models (GPT-4.1, Claude 4, Gemini 2.5, Qwen3) and found accuracy degrades well before the window fills, often around 300k to 400k tokens on a model that nominally takes a million. Stuffing more in doesn't make the model read more carefully; it gives it more to get lost in. It also does nothing about the core problem, which is that you'd be pasting in the wrong document.
"Our chunking is naive. We should switch to semantic chunking." Also a distraction. Several benchmarks from 2025 and 2026 show plain recursive or fixed size chunking matching or beating fancy semantic chunking, with the result driven far more by domain and choice of embedding model than by the chunker anyone agonized over. I'd call chunking obsession cargo culting. The team spent a sprint on it and moved recall by nothing, because a well chunked stale document is still stale.
"Let's make it agentic. Let it search in rounds until it's confident." Agentic retrieval does help on multi hop questions. Bolt it onto a rotting index, though, and you've built a more expensive, slower way to retrieve the same wrong document three times before citing it. If ingestion and evaluation are broken, autonomy only makes the error cost more.
Agentic loops on a stale index don't find the truth. They find the same wrong answer, slower, and bill you for the effort.
None of these were stupid ideas. They were the right tools aimed at the wrong failure. Hybrid search, reranking, GraphRAG and the rest of the family are real and worth knowing, and I've laid them out in a plain language guide to the forms of RAG. That article helps you choose the tool. This one is about a tool you chose correctly in March that stops working by June while your dashboard stays green.
Underneath the whole project sat a misconception that set up the disappointment: the belief that RAG eliminates hallucination.
It reduces it, and then it moves it somewhere else. When retrieval hands the model conflicting or outdated context, you get a grounded, cited, confident, wrong answer. That's more dangerous than an ungrounded guess, because the citation makes it look verified. Stanford's audit of commercial legal RAG tools gives the number to remember: even the best purpose built systems still hallucinated between 17% and 33% of the time, including fabricated cases and sycophantic agreement with a user's false premise. General GPT-4 hit 43%. Grounding narrows the gap without closing it.
Sell RAG as a hallucination cure and the week three letdown is guaranteed. For the longer version of that argument, we've written separately about why AI hallucinations are a business risk you manage rather than a bug you fix. In short, a system that can be confidently wrong has to be watched all the time, not certified once and trusted forever.
What would have kept this assistant alive? Neither a better model nor a cleverer retriever. It needed operational discipline, the dull kind that doesn't demo well, in three specific places.
First, retrieval quality has to be in your CI, not your quarterly review. The tooling exists and is mostly open source: RAGAS for fast checks without reference answers during development, DeepEval as a pytest style gate that fails the build when recall or faithfulness drops below threshold, TruLens or Arize Phoenix for tracing the same signals in production. A change to the corpus, the chunker or the embedding model should trip a regression test the same way a broken unit test does. If retrieval precision at 5 isn't a number your pipeline enforces on every deploy, you are flying on feel.
Second, drift needs monitoring as a signal of its own. Track recall and answer relevance over time on a fixed golden set of real questions, and alarm on the slope, not only the absolute value. A drop from 0.92 to 0.74 is invisible in a snapshot and screaming in a trend. That's as much a KPI question as an engineering one, and choosing metrics that map to business outcomes instead of vanity is a discipline we've written about here.
Third, reindex cadence is your accuracy ceiling, so set it per corpus. Volatile content (fee schedules, product terms, rate tables) gets a short time to live and frequent re-embedding, and stable content can ride. When a document is superseded, delete the old vectors instead of adding the new ones next to them, or you've left the corpse in the index to keep outranking its replacement.
The good news is that investment on the retrieval side pays off well. Anthropic's Contextual Retrieval work showed that contextualizing chunks before embedding, combining vector and keyword search and adding a reranker cut retrieval failures by up to 67%, at a contextualization cost of about a dollar per million document tokens with prompt caching. A real, cheap, measurable win. It is also a one time architecture improvement. It buys you a higher starting recall and does nothing to stop that recall from sliding once your corpus starts changing, which happens from day one.
The model never killed this assistant. An ops gap dressed up as an ML problem did.
On day one the assistant did exactly what it was built to do, and it kept doing it, fast and fluent, long after the ground under it had moved. Nothing crashed because nothing was designed to crash. RAG fails as a slow leak, not a break, and slow leaks don't make noise.
Choosing among the forms of RAG is, by now, a solved enough problem. Keeping them alive is the actual job, meaning reindex cadence, drift monitoring, retrieval quality in CI, and owning the pipeline as infrastructure instead of as a prompt someone wrote once. Less exciting than a slide full of arrows, sure. It's also what separates a system that works on Monday morning, in production, with real staff and real customers, from one that's misleading everybody by week three while the dashboard stays a reassuring green.
We do this instrumentation for a living, mostly for banks and public sector teams across the Middle East, Africa and Europe who can't afford a confidently wrong answer on a live account. If your RAG has been "kind of working but not really" lately, you're probably breathing the carbon monoxide already. Go measure it before your users do.
Len
Updated on 12 October 2026.
