The famous failure stat is making the rounds again. This time it's in Linas Beliūnas's newsletter, under a headline I'd happily frame and hang on the office wall: 95 percent of bank AI pilots fail because chatbots can't run compliance workflows. The number goes back to MIT's GenAI Divide report and has been picked up across banking research since. I've quoted it myself more than once. Guilty.
What's new is the diagnosis, and for once nobody blames the model or the data. They blame the unit of work. A chatbot automates conversations, while a bank runs on cases.
A conversation is a session. It opens when the customer types, it answers, it closes when the customer leaves, and it remembers nothing. Perfect for "what are your opening hours." Useless for almost everything a bank actually gets paid to do.
Think about what real banking work looks like. An onboarding is a document collection that runs for days and ends in a compliance decision. A collections arrangement is a promise you have to track for three months. A KYC review touches the core, the screening provider, the document store and a human approver. All of it runs long, crosses systems and has to end in something an auditor can reconstruct.
Take one business account opening and you'll see it. On day zero the owner of a small company starts an application in chat, which is the conversational part and takes about six minutes. On day one the registry check and sanctions screening run, and the bank asks for proof of who actually owns the company. The documents come in on day three and get checked, and one passport scan turns out to be expired. By day eight nobody has uploaded a new one, and the application is about to lapse whether anyone types or not. Day twelve brings the decision, an account opened in the core, and a record the compliance team may one day have to show a supervisor.
That's one case, five systems and a dozen state changes, and exactly one of those steps looks like a conversation. Automate those first six minutes brilliantly and you've automated the doorbell of a house you never entered.

Forrester explained why better than anyone this year: a long running agent doesn't behave like a chatbot, it behaves like a distributed system. Distributed systems need orchestration, identity and context discipline that most companies have never built. That's the 95 percent in one sentence. The pilots that die automated the conversation, and the ones that survive automate the outcome.
The market is admitting the same thing, if you read the fine print. Intercom builds one of the best horizontal support agents out there, and it offers a money back guarantee on its Fin agent: a million dollars if you don't reach a 65 percent resolution rate. Intercom itself describes that floor as the resolution rate of humans. Klarna's famous assistant handled two thirds of its support chats in month one. Two thirds, 65 percent. The best conversational AI in production keeps landing in the same band.
I don't think that's a coincidence, and I don't think it's a model problem. It's simply the share of work that actually is conversational. The last third is made of disputes, onboardings and arrangements, in other words cases, and you can't deflect a case because it was never a ticket to begin with. What's left is work that carries state, and no amount of prompt engineering turns a goldfish into a filing cabinet.
We see this every day in our own delivery. The onboarding workflow we built for document collection keeps track of every candidate in a database: which documents arrived, which were rejected and why, when the last reminder went out, when to escalate. The friendly chat that asks for a passport photo is the small bit. Most of the build lives in state, deadlines and write backs. That fits with what the newsletter reports, namely that specialist banking AI teams spend 80 percent of their engineering on compliance and integration. It also mentions a UK card provider that rebuilt its disputes process around a case running agent. Resolution went from five to seven days down to two to three, and first contact evidence rates went from under 25 percent to 80. Same picture on every case shaped project we've shipped.
So if cases are where the value is, why does every pilot start with a session bot?
Because pilots get picked on how well they demo. A session fits in a steering meeting (ask a question, watch a beautiful answer, approve the budget). A case takes twelve days to show, and most of those days look like nothing is happening, which nobody wants to put on a slide. I've sat in that steering meeting more times than I can count.
So companies keep funding the work that fits the meeting instead of the work that carries the money, and then file the result under "AI didn't deliver." A big part of the 95 percent is a buying bias. We bought what demos in minutes to automate work that lives in weeks.
Even when a pilot picks the right unit, there's a second brake. I see almost every project slow down on stakeholder management. If I break down where the time really goes, it's roughly 20 percent technology, 20 percent customization and 60 percent approvals and stakeholders. I'm not complaining about that, it's how large organizations work, and a case touches more of them than a chatbot ever will: risk, compliance, legal, IT, the business owner.
20 percent technology, 20 percent customization, 60 percent approvals. Plan for the 60.
Over the years we've become experts in that 60 percent, and we know how to fast track it. Almost nobody puts it in the pilot plan, though, which is why so many pilots never make it past the steering meeting.
Strip it down and a case running agent needs four things a session bot lacks. First, memory that survives the conversation, so state in a database and not in a prompt. Then the ability to act across systems: read the core, write the ticket, call the screening API. It also needs to know who may decide what, so the agent drafts and a named human approves. And it needs a clock.
The clock is the most underrated of the four. Cases have deadlines, waiting periods and follow ups. A session bot experiences time as one conversation, a case runner experiences it as a calendar.
Paul is writing a deeper piece on what that coordination layer looks like in engineering terms, so I'll leave the architecture to him. The strategic point is simple, and I'll keep repeating it: "which model do you use" is still the wrong first question. Models are interchangeable inside a case runner, the machinery around them much less so.
There's a test so simple you can run it in the steering meeting. Ask what your AI does on day eight.
An application started on day zero. The owner hasn't replied since day three, when the bank asked for a valid passport, and on day eight the application lapses in seventy two hours. Does the AI send a reminder, alert the relationship manager, flag the file? Or does it sit there, very smart and completely idle, waiting for someone to type?
If your AI only acts when someone messages it, you built a chatbot. If it acts because a deadline arrived, you built a case runner.
None of this needs a better model. All of it needs state, timers, integrations and authority, the plain machinery that separates the five percent from the ninety five. So before the next pilot gets approved, change the demo script. Don't ask the vendor for a beautiful answer, ask them to show you day eight.
And if your onboardings, disputes or collections still run on an AI with the memory of a goldfish, I'm always happy to compare notes.
Updated on 12 October 2026.