AI Technology06-10-20267 min read

A Conversation Is Not a Case

The 95 percent pilot failure stat is making the rounds again, this time with a diagnosis I actually agree with: chatbots automate sessions, but a bank runs cases. Why the unit of work decides whether your AI survives production, and the day eight test that shows which one you built.

Len Debets
Len Debets
CTO & Co-Founder
A Conversation Is Not a Case

The famous failure stat is making the rounds again. This time it's in Linas Beliūnas's newsletter, under a headline I'd happily frame and hang on the office wall: 95 percent of bank AI pilots fail because chatbots can't run compliance workflows. The number goes back to MIT's GenAI Divide report and has been picked up across banking research since. I've quoted it myself more than once. Guilty.

What's new is the diagnosis, and for once nobody blames the model or the data. They blame the unit of work. A chatbot automates conversations, while a bank runs on cases.

Two different things wearing the same interface

A conversation is a session. It opens when the customer types, it answers, it closes when the customer leaves, and it remembers nothing. Perfect for "what are your opening hours." Useless for almost everything a bank actually gets paid to do.

Think about what real banking work looks like. An onboarding is a document collection that runs for days and ends in a compliance decision. A collections arrangement is a promise you have to track for three months. A KYC review touches the core, the screening provider, the document store and a human approver. All of it runs long, crosses systems and has to end in something an auditor can reconstruct.

Take one business account opening and you'll see it. On day zero the owner of a small company starts an application in chat, which is the conversational part and takes about six minutes. On day one the registry check and sanctions screening run, and the bank asks for proof of who actually owns the company. The documents come in on day three and get checked, and one passport scan turns out to be expired. By day eight nobody has uploaded a new one, and the application is about to lapse whether anyone types or not. Day twelve brings the decision, an account opened in the core, and a record the compliance team may one day have to show a supervisor.

That's one case, five systems and a dozen state changes, and exactly one of those steps looks like a conversation. Automate those first six minutes brilliantly and you've automated the doorbell of a house you never entered.

The unit of work is the whole argument

Forrester explained why better than anyone this year: a long running agent doesn't behave like a chatbot, it behaves like a distributed system. Distributed systems need orchestration, identity and context discipline that most companies have never built. That's the 95 percent in one sentence. The pilots that die automated the conversation, and the ones that survive automate the outcome.

The plateau is not an accident

The market is admitting the same thing, if you read the fine print. Intercom builds one of the best horizontal support agents out there, and it offers a money back guarantee on its Fin agent: a million dollars if you don't reach a 65 percent resolution rate. Intercom itself describes that floor as the resolution rate of humans. Klarna's famous assistant handled two thirds of its support chats in month one. Two thirds, 65 percent. The best conversational AI in production keeps landing in the same band.

I don't think that's a coincidence, and I don't think it's a model problem. It's simply the share of work that actually is conversational. The last third is made of disputes, onboardings and arrangements, in other words cases, and you can't deflect a case because it was never a ticket to begin with. What's left is work that carries state, and no amount of prompt engineering turns a goldfish into a filing cabinet.

We see this every day in our own delivery. The onboarding workflow we built for document collection keeps track of every candidate in a database: which documents arrived, which were rejected and why, when the last reminder went out, when to escalate. The friendly chat that asks for a passport photo is the small bit. Most of the build lives in state, deadlines and write backs. That fits with what the newsletter reports, namely that specialist banking AI teams spend 80 percent of their engineering on compliance and integration. It also mentions a UK card provider that rebuilt its disputes process around a case running agent. Resolution went from five to seven days down to two to three, and first contact evidence rates went from under 25 percent to 80. Same picture on every case shaped project we've shipped.

Why pilots keep picking the wrong unit

So if cases are where the value is, why does every pilot start with a session bot?

Because pilots get picked on how well they demo. A session fits in a steering meeting (ask a question, watch a beautiful answer, approve the budget). A case takes twelve days to show, and most of those days look like nothing is happening, which nobody wants to put on a slide. I've sat in that steering meeting more times than I can count.

So companies keep funding the work that fits the meeting instead of the work that carries the money, and then file the result under "AI didn't deliver." A big part of the 95 percent is a buying bias. We bought what demos in minutes to automate work that lives in weeks.

Even when a pilot picks the right unit, there's a second brake. I see almost every project slow down on stakeholder management. If I break down where the time really goes, it's roughly 20 percent technology, 20 percent customization and 60 percent approvals and stakeholders. I'm not complaining about that, it's how large organizations work, and a case touches more of them than a chatbot ever will: risk, compliance, legal, IT, the business owner.

20 percent technology, 20 percent customization, 60 percent approvals. Plan for the 60.

Over the years we've become experts in that 60 percent, and we know how to fast track it. Almost nobody puts it in the pilot plan, though, which is why so many pilots never make it past the steering meeting.

What a case runner has that a chatbot doesn't

Strip it down and a case running agent needs four things a session bot lacks. First, memory that survives the conversation, so state in a database and not in a prompt. Then the ability to act across systems: read the core, write the ticket, call the screening API. It also needs to know who may decide what, so the agent drafts and a named human approves. And it needs a clock.

The clock is the most underrated of the four. Cases have deadlines, waiting periods and follow ups. A session bot experiences time as one conversation, a case runner experiences it as a calendar.

Paul is writing a deeper piece on what that coordination layer looks like in engineering terms, so I'll leave the architecture to him. The strategic point is simple, and I'll keep repeating it: "which model do you use" is still the wrong first question. Models are interchangeable inside a case runner, the machinery around them much less so.

The day eight test

There's a test so simple you can run it in the steering meeting. Ask what your AI does on day eight.

An application started on day zero. The owner hasn't replied since day three, when the bank asked for a valid passport, and on day eight the application lapses in seventy two hours. Does the AI send a reminder, alert the relationship manager, flag the file? Or does it sit there, very smart and completely idle, waiting for someone to type?

If your AI only acts when someone messages it, you built a chatbot. If it acts because a deadline arrived, you built a case runner.

None of this needs a better model. All of it needs state, timers, integrations and authority, the plain machinery that separates the five percent from the ninety five. So before the next pilot gets approved, change the demo script. Don't ask the vendor for a beautiful answer, ask them to show you day eight.

And if your onboardings, disputes or collections still run on an AI with the memory of a goldfish, I'm always happy to compare notes.

Updated on 12 October 2026.

Len Debets
Len Debets
CTO & Co-Founder
Writes about Enterprise Architecture, LLM, IT Security, RAG Systems
Len Debets on LinkedIn
Published on 06-10-2026

Related Articles

9 Things I Really Hate About AI
AI Technology12-05-2025

9 Things I Really Hate About AI

I think it's great that technology is so embedded in our daily lives. It helps us get knowledge faster, finish tasks more efficiently, gives us inspiration, and occasionally scares the hell out of us with those crazy (fake) videos. I help a lot of companies implement AI, so in the end it pays my bills. But after spending a ridiculous amount of time with all these new technologies, it's time to write down the things I really hate about AI.

Read More →
Agentic AI Languages and Dialects: Why Voice Quality Is Still the Hard Part
AI Technology10-04-2026

Agentic AI Languages and Dialects: Why Voice Quality Is Still the Hard Part

Agentic systems promise autonomous workflows, but speech and dialect quality fail first. Why Arabic, Turkish, Chinese and more run into the same gap.

Read More →
Agentic AI vs Generative AI: What Changes When AI Starts Doing the Work
AI Technology23-10-2026

Agentic AI vs Generative AI: What Changes When AI Starts Doing the Work

Generative AI writes. Agentic AI does. That sounds like a small step, but it changes what can go wrong, what you need to govern and how you measure success. The real difference between agentic AI and generative AI, why most agentic projects stall, and what the ones in production have in common.

Read More →

Stay Updated

New posts on conversational AI, enterprise automation and customer experience, sent to your inbox when they go live

No spam, unsubscribe at any time