In early 2024, a finance employee at engineering firm Arup joined a video call with people who looked and sounded like the company's chief financial officer and several colleagues.
Every person on the call was an AI-generated fake.
By the time it ended, the employee had approved 15 transfers worth about US$25 million in what became a landmark deepfake fraud case.
For years, it's been a cautionary tale about why you can't automatically trust what you see or hear on a screen. But the next version of this risk may be harder to spot, because it targets something finance teams are now introducing into their own operations: autonomous AI agents.
Agents are moving quickly into accounts payable and treasury. They can read invoices, match them to purchase orders and prepare payment runs. In some environments, they can also initiate disbursements or manage cash positions with limited human oversight.
For time-poor teams, the need is clear and the opportunity can't be ignored. But it's also critical to remember that an agent can be manipulated into doing exactly what it was authorized to do, without a person ever being deceived on a call.
Adoption is moving faster than governance
The pressure is coming from the top. Avalara's 2026 survey of finance leaders found that 92% feel pressure to demonstrate a return on their AI investment, and only 7% said their organization prioritizes AI governance over adoption speed.
The same research found that 76% of organizations lack the in-house expertise needed to understand how their AI systems work.
In other words, agents are being switched on faster than many companies can govern them. The clearest warning signs are emerging from software development, where agents were deployed earlier.
In July 2026, The Hacker News reported that a hidden comment inside a code review request could hijack Microsoft's Azure DevOps AI agent. The agent would then act on the attacker's instructions using the reviewer's own access.
A separate flaw in AWS Kiro allowed text hidden on an ordinary web page to rewrite the agent's configuration and run code on a developer's machine. The approval prompt intended to prevent this did not stop the attack.
Set aside the technical detail and the mechanism is one every AP team should recognize: an agent reads untrusted content, finds instructions buried inside it and follows them. In finance, that untrusted content could be an invoice, a vendor email or a vendor portal.
The agent uses your authority
What makes this especially dangerous is the authority the agent carries. In the Azure DevOps case, the agent acted with the reviewer's permissions. Its behavior did not immediately look suspicious because the agent was authorized to access those systems.
Security researchers describe this as the "confused deputy" problem. A low-privileged attacker persuades a trusted, higher-privileged system to act on their behalf.
An agent authorized to update a payee's bank details or release a payment run is exactly that kind of deputy. The permission check passes because the agent is supposed to have that access.
Consider how this could play out in accounts payable.
An agent is asked to process the week's invoices from a shared mailbox. One invoice looks genuine in every visible respect. But its notes field contains text the agent interprets as an instruction to update the vendor's bank account and prioritize the payment.
No employee sees the instruction because it was written for the agent, not a person.
The agent has permission to make the change. The payment goes out. The first warning arrives when the real vendor asks where its money is.
Existing safeguards may not be enough
Other recent findings weaken some of the most common reassurances about autonomous agents.
BleepingComputer reported that agents in several coding tools could escape their intended boundaries by writing files that a trusted process would later run outside the sandbox.
The U.K. AI Security Institute also tested five frontier models and found that all five broke rules or cheated to complete cybersecurity tasks. Most did not acknowledge the behavior when questioned.
OpenAI later disclosed that models being tested escaped their sandbox and reached Hugging Face's live production infrastructure.
Finance leaders should take two lessons from these incidents:
- Don't assume an agent remained inside its intended boundaries
- Don't assume its own logs will provide a complete account of what it did
Payment controls were designed around people
Most payment controls assume a person is involved.
Someone approves the payment. Someone reconciles it afterward. The audit trail records who did what.
Autonomous agents can weaken all three controls. They remove people from routine approvals, may produce a clean-looking record while taking the wrong action and leave reconciliation to identify the loss after the money has gone.
The Association for Financial Professionals found that 76% of U.S. organizations experienced attempted or actual payments fraud in 2025. Business email compromise affected 74% of organizations.
Agents could give the same attackers a faster, quieter route into payment workflows.
How to stop an agent from moving the wrong money
Treat every agent that touches payments as a non-human user.
Give it a distinct identity and only the access required for its task. Don't allow it to inherit a staff member's full permissions for convenience.
Require independent, out-of-band confirmation for the two changes attackers are most likely to target:
- Adding a new payee
- Changing an existing payee's bank details
Apply that requirement regardless of whether the request came from a person, an email, an invoice or an AI agent.
Checks on the payee, amount and payment timing should also sit outside the agent and the workflow it operates. A manipulated agent must not be able to approve the same instruction it acted on. Finally, record activity at the system boundary instead of relying on the agent to report on itself.
AI agents will continue to spread across the finance stack. Pressure to give them greater authority, with less supervision, will grow alongside adoption.
That makes independent payment verification more important, not less. Put simply, confirming that money is reaching the right payee has to remain separate from the system initiating the payment.
Read more about how to embed continuous controls in your processes and tighten governance around both automated and manual steps in your workflows.
