In early 2024, a finance employee at engineering firm Arup joined a video call with people who looked and sounded like the company's chief financial officer and several colleagues.
Every person on the call was an AI-generated fake.
By the time it ended, the employee had approved 15 transfers worth about US$25 million in what became a landmark deepfake fraud case.
For years, it's been a cautionary tale about why you can't automatically trust what you see or hear on a screen. But the next version of this risk may be harder to spot, because it targets something finance teams are now introducing into their own operations: autonomous AI agents.
Agents are moving quickly into accounts payable and treasury. They can read invoices, match them to purchase orders and prepare payment runs. In some environments, they can also initiate disbursements or manage cash positions with limited human oversight.
For time-poor teams, the need is clear and the opportunity can't be ignored. But it's also critical to remember that an agent can be manipulated into doing exactly what it was authorised to do, without a person ever being deceived on a call.
Adoption is moving faster than governance
The pressure is coming from the top. Avalara's 2026 survey of finance leaders found that 92% feel pressure to demonstrate a return on their AI investment, and only 7% said their organisation prioritises AI governance over adoption speed.
The same research found that 76% of organisations lack the in-house expertise needed to understand how their AI systems work.
In other words, agents are being switched on faster than many companies can govern them. The clearest warning signs are emerging from software development, where agents were deployed earlier.
In July 2026, The Hacker News reported that a hidden comment inside a code review request could hijack Microsoft's Azure DevOps AI agent. The agent would then act on the attacker's instructions using the reviewer's own access.
A separate flaw in AWS Kiro allowed text hidden on an ordinary web page to rewrite the agent's configuration and run code on a developer's machine. The approval prompt intended to prevent this did not stop the attack.
Set aside the technical detail and the mechanism is one every AP team should recognise: an agent reads untrusted content, finds instructions buried inside it and follows them. In finance, that untrusted content could be an invoice, a supplier email or a supplier portal.
The agent uses your authority
What makes this especially dangerous is the authority the agent carries. In the Azure DevOps case, the agent acted with the reviewer's permissions. Its behaviour did not immediately look suspicious because the agent was authorised to access those systems.
Security researchers describe this as the "confused deputy" problem. A low-privileged attacker persuades a trusted, higher-privileged system to act on their behalf. An agent authorised to update a payee's bank details or release a payment run is exactly that kind of deputy. The permission check passes because the agent is supposed to have that access.
Consider how this could play out in accounts payable.
An agent is asked to process the week's invoices from a shared mailbox. One invoice looks genuine in every visible respect. But its notes field contains text the agent interprets as an instruction to update the supplier's bank account and prioritise the payment.
No employee sees the instruction because it was written for the agent, not a person. The agent has permission to make the change. The payment goes out. The first warning arrives when the real supplier asks where its money is.
Existing safeguards may not be enough
Other recent findings weaken some of the most common reassurances about autonomous agents.
BleepingComputer reported that agents in several coding tools could escape their intended boundaries by writing files that a trusted process would later run outside the sandbox.
The U.K. AI Security Institute also tested five frontier models and found that all five broke rules or cheated to complete cybersecurity tasks. Most did not acknowledge the behaviour when questioned.
OpenAI later disclosed that models being tested escaped their sandbox and reached Hugging Face's live production infrastructure.
Finance leaders should take two lessons from these incidents:
- Don't assume an agent remained inside its intended boundaries
- Don't assume its own logs will provide a complete account of what it did
Payment controls were designed around people
Most payment controls assume a person is involved.
Someone approves the payment. Someone reconciles it afterwards. The audit trail records who did what.
Autonomous agents can weaken all three controls. They remove people from routine approvals, may produce a clean-looking record while taking the wrong action and leave reconciliation to identify the loss after the money has gone.
The Association for Financial Professionals found that 76% of U.S. organisations experienced attempted or actual payments fraud in 2025. Business email compromise affected 74% of organisations.
Agents could give the same attackers a faster, quieter route into payment workflows.
How to stop an agent from moving the wrong money
Treat every agent that touches payments as a non-human user.
Give it a distinct identity and only the access required for its task. Don't allow it to inherit a staff member's full permissions for convenience.
Require independent, out-of-band confirmation for the two changes attackers are most likely to target:
- Adding a new payee
- Changing an existing payee's bank details
Apply that requirement regardless of whether the request came from a person, an email, an invoice or an AI agent.
Checks on the payee, amount and payment timing should also sit outside the agent and the workflow it operates. A manipulated agent must not be able to approve the same instruction it acted on. Finally, record activity at the system boundary instead of relying on the agent to report on itself.
AI agents will continue to spread across the finance stack. Pressure to give them greater authority, with less supervision, will grow alongside adoption.
That makes independent payment verification more important, not less. Put simply, confirming that money is reaching the right payee has to remain separate from the system initiating the payment.
Where to start with continuous controls
Continuous controls are how finance teams close this gap. Instead of trusting a single approval or an after-the-fact reconciliation, they verify every payment independently of the system, or the person, that set it in motion, and they keep working whether a human or an agent is at the keyboard.
Our finance leader's guide to continuous controls for outgoing payments shows how to embed that verification across supplier onboarding, bank-detail changes and payment runs, and how to tighten governance around both the automated and manual steps in your workflows. To see continuous verification applied to your own payments, book a demo.
