Let me start by saying this is not an argument against AI agents. The gains in BFSI are real. At Bajaj General Insurance, an AI agent checking motor claims against policy terms cut claims admissibility time by 60%, according to Bajaj Finserv’s Q2 FY27 disclosures. Customers get faster decisions, consistent rules and answers at any hour.

The standard safeguard is human oversight. A person reviews what the agent does. And here is the problem I keep seeing: human oversight is not a fixed control. It decays.

How oversight fades

In the first weeks after an agent goes live, reviewers check every decision carefully. By the third month, the agent has been right most of the time, so they skim. By the sixth month, volumes have grown and the queue never empties, so they approve in bulk. A year in, review is a step in the workflow rather than a judgement.

Nobody is being careless. People naturally trust a system that is usually right. Review is the first thing to be squeezed on a busy day. And customers who are wrongly rejected rarely complain, so nothing on the dashboard looks wrong.

The clearest picture of where this ends comes from US health insurance. ProPublica reported that Cigna’s doctors denied more than 300,000 claims in two months using an automated review system, spending an average of 1.2 seconds per claim. Cigna disputes the characterisation, but 1.2 seconds is what fully decayed oversight looks like: a human signature on a machine’s decision.

Why it matters more with AI

Decay would be less worrying if AI agents only made rare, random mistakes. They don’t. They faithfully apply whatever a person has set up, including that person’s errors.

A threshold typed as 0.5 instead of 0.05. Last year’s policy document left in the agent’s knowledge base. A field mapped to the wrong column. These are ordinary slips anyone makes on a busy afternoon. An employee who makes one affects a single customer. An agent applies it to every case it touches, without fatigue and without doubt.

The agent then becomes either too strict or too lenient. Too strict, and it rejects good customers: KYC files bounced for a minor name mismatch, genuine transactions frozen, valid claims denied. Too lenient, and it promises what the institution never authorised. In 2024, a Canadian tribunal held Air Canada liable after its chatbot told a grieving customer about a refund the airline’s policy didn’t allow. The tribunal was clear that a company owns what its AI says.

Australia’s Robodebt scheme shows how far a single automated rule can go when oversight fails. A flawed income-averaging method raised around A$1.76 billion in unlawful debts against roughly 433,000 people, and ran for years before it was stopped.

Put simply: AI agents turn small human errors into large ones, and decayed oversight means nobody catches them.

Designing oversight that doesn’t depend on attention

If human attention fades, asking people to pay more attention won’t fix it. The answer is to build oversight into the system, so it keeps working on the day people are busy, tired or too trusting.

In practice, that comes down to a few disciplines. Every change to an agent’s rules, prompts or data needs a second approver and a test on past cases before release, because that is where most errors enter. New rules run quietly alongside people first, then on a small share of cases, so a mistake reaches fifty customers rather than fifty thousand. Agents never make an adverse final decision about a customer on their own.

Then the system watches itself. Approval and rejection rates trigger alarms when they shift. A sample of decisions is reviewed every day by someone independent, chosen automatically rather than when someone remembers. Review time and override rates are tracked, because when review takes seconds and overrides drop to zero, oversight has decayed and the data should say so. And every customer complaint about an AI decision goes straight back to the team that owns the agent.

Regulators are heading the same way. RBI’s April 2026 advisory asks institutions to control AI agents’ access and third-party AI risk, and CERT-In warned in October 2026 that AI agents had reached organisations’ systems without anyone noticing. “We have people reviewing it” will not satisfy a regulator for long. Evidence that oversight actually works will.

The question for leaders

The question I would put to any BFSI leadership team is simple: if someone entered a wrong threshold in one of your AI agents this morning, how quickly would you know?

If the honest answer is “when the complaints arrive”, oversight has not been designed in.

Human oversight of AI agents doesn’t fail suddenly. It fades, one busy day and one trusted decision at a time. The institutions that use AI well won’t be the ones with the most reviewers. They’ll be the ones whose systems keep customers safe on the day those reviewers stop looking.

At Venturesathi, that is how we build AI into the regulated operations we run for banks, NBFCs, fintechs and insurers. If it’s the standard you want, we can help.