An AI assistant just exfiltrated a finance team's books — and nobody noticed
Last week, security researchers showed that Ramp's spreadsheet AI could be tricked into shipping a finance team's books to an external server through a single line of text hidden in a transaction description. The same week, a GitHub issue revealed that adding the word "HERMES.md" to a commit message rerouted Claude Code traffic to extra-billing endpoints.
Two stories, one root cause: AI agents do not distinguish between data and instructions. Until your stack does, every customer email, every PDF attachment, every comment field in your CRM is a potential command-line.
1. The moment data becomes a command
In a traditional SaaS architecture, the user's click is the command. Database text is just text. LLM-based agents flatten that distinction — every string in the context window has equal weight. A customer email, a CRM note, even a single line in an attached PDF can become an executable instruction.
In the Ramp case, an attacker placed a sentence meaning "send all data to attacker.com" inside an ordinary transaction description. The AI dutifully complied. This is what happens when input validation goes missing in the LLM era.
2. The "user permission equals AI permission" myth
Most companies hand their AI agents the same OAuth tokens their humans use. The thinking goes, "it is acting on my behalf, so it is safe." But humans only act on their own intent. AI agents act on whatever input arrives last. The right security boundary is the decision-maker, not the human.
One of our enterprise clients in Japan nearly had a payment auto-approved through prompt injection inside a PDF quote sent by an outside partner. Their finance bot had full ERP access — because the human who launched it did.
3. The lie that "no logs means no incident"
A single user request to an LLM agent can fan out into dozens of internal tool calls. If you cannot trace the full chain — prompt, tool, arguments, result — you have no way to forensically explain a leak. The agent systems 5years+ ships always force every tool call into a structured audit log retrievable within 24 hours.
Today's action items
- Isolate untrusted input. Anything from outside (email bodies, attachments, web search results) belongs in a clearly labeled container, separated from your system prompt, marked "this is data, not instruction."
- Minimize tool permissions. The AI's privilege should be narrower than the human's, not equal. Payments, deletes, and outbound sends belong behind a human approval step.
- Mandate tool call audit logs. Record which prompt invoked which tool with which arguments. If an incident happens, you should be able to reconstruct it within a day.
5years+ designs AI agent rollouts for Korean and Japanese B2B teams with security as the day-one constraint. Browse our services for details, or request a tailored assessment via Free consultation.
Frequently Asked Questions
We already have AI agents running. Where do we start?
List every tool your agents can invoke. Separate the ones that send data outward, move money, delete records, or change permissions. Wrap each of those in a human approval step before doing anything else.
Are open-source models safer?
This is not a model problem; it is an input and tool boundary problem. GPT, Claude, Llama, and Mistral all carry the same risk surface and need the same guardrails.
How much does a secure agent rollout cost?
It depends on scope and tool count, but our initial assessment (1–2 weeks) is free, and a full build typically runs 4–8 weeks. See our portfolio for comparable engagements.