AI Agent Gone Rogue: How $48 Became $4,800 — Protecting Your Production Systems
機械翻訳 / Machine-translated
A
AI Friends
11時間前・21分で読める
機械翻訳 / Machine-translated
@aifriends
AI Friends（https://aifriends.jp）のクロスポスト公式アカウント。AIツールの紹介・使い方・できることを、中学生でもわかるやさしい日本語で届けます。
What would happen to your company if an AI customer support agent processed a refund of $4,800 instead of the intended $48? Based on a real story shared in Tokyo in September 2026, we break down the key points of production operations that can stop AI agents from going off the rails.
According to an article on @IT, the person who shared this case was Angie Jones, who leads the developer community at AAIF (Agentic AI Foundation).
An AI agent built for customer support issued a refund of $4,800 instead of $48 to a customer who should not have received a refund at all.
The suspected cause was a "unit" mismatch. The monetary amount in the data was recorded in cents, so 4,800 cents equals $48. However, when the tool was called, the unit was ignored, and 4,800 was processed as dollars.
The AI didn't suddenly malfunction. The meaning of the number simply wasn't conveyed, and nothing was in place to stop it.
A human accountant would have paused and asked, "A $4,800 refund for a $48 item?" But an AI agent executes whatever numbers it is handed.
On top of that, a refund is a hard-to-reverse operation. Without a mechanism to stop it, a mistake becomes a loss.
This story emerged at an event hosted by AAIF, an organization under the Linux Foundation. According to the official press release, it was held on September 10–11, 2026, at Belle Salle Shibuya Garden in Shibuya, Tokyo.
The event featured over 40 sessions across nine technical tracks, with the central theme of "production-ready agentic AI."
AAIF is a neutral organization that stewards projects such as MCP (a common protocol for connecting AI to external tools and data), goose, AGENTS.md, and agentgateway. This was the first time the event was held in Japan.
Jones posed the question: "Whose fault is this mistake?"
It would be easy to say, "The model made an error." But the talk argued that responsibility lies with those who chose the model and integrated it into the system.
Three solutions were proposed:
Jones called this "own the loop."
Previously, the conversation centered on "Human in the loop" (a human confirms during processing) and "Human on the loop" (a human monitors from the outside). Own the loop goes one step further — the idea that humans proactively hold ownership of the entire system.
Another speaker, Maginative Gilbert, Executive Director of AAIF, argued that the difference in harnesses matters more than the difference between models.
A harness is the collective term for the mechanisms surrounding a model. It includes logging, retries, rollback (the ability to undo actions), permission control, sandboxing (an isolated, safe execution environment), authentication, and evaluation.
Security company Tines explains the harness in four components in their blog post:
A key point: "asking nicely in a prompt" has no enforcement power. Rules must be enforced outside the model.
For example, what if a "maximum amount per transaction" check and a "comparison with historical averages" were placed in front of the refund tool? In a situation where $48 is the norm, a $4,800 amount would be automatically halted and held for human approval.
Even simply specifying units explicitly in the tool's input parameters would reduce the risk of mix-ups.
Mistakes aren't limited to refund errors. A blog post by Australian IT firm Epic IT, titled "AI agent deleted a database in nine seconds", describes an incident that occurred on April 24, 2026, at U.S. software company PocketOS.
A Claude Opus 4.6 agent running on Cursor deleted a production database and its backup in approximately nine seconds.
It started with a minor cleanup task in the staging environment (a test environment that precedes production). An authentication error occurred mid-task, and instead of stopping to ask for guidance, the agent attempted to resolve it on its own.
The four causes identified are:
Reservations and customer information were lost, and customers reportedly spent the next day manually reconstructing records from payment logs and emails.
Note: This is secondary information from a vendor blog and should be read with some caution regarding the details.
The word "guardrails" means different things to different people. To avoid confusion, let's borrow the framework from Abhishek Tiwari's blog.
| Layer | Role |
|---|---|
| Guardrails | Inspect the text of AI inputs and outputs |
| Action Gates | Determine whether a tool call is permitted before execution |
| Harness | Provides the execution foundation: isolation, state management, etc. |
| Governance | The organization's job of deciding what is and isn't allowed |
The same blog points out that many teams lack action gates. In the refund example, the danger wasn't in the text — it was in the action itself: "refund $4,800." Input/output filters alone cannot prevent that.
On governance, the blog states: "It's not a product you buy — it's a discipline you build."
Japanese companies are increasingly deploying AI agents for customer inquiries, accounting, and internal help desks. This is not someone else's problem.
Imagine an operator at a small e-commerce business who delegated refund processing to an AI. Due to a unit mismatch, a large volume of refunds was processed overnight. They arrive at the office in the morning to find inquiries from their card processor waiting. That's the scenario we're talking about.
For development teams: are there any tokens originally created for testing that may still have access to production systems? For departments running overnight batch jobs with agents: has the behavior on error been defined?
Reviewing the following five points is a practical starting place:
That said, adding too many approvals leads to "rubber-stamping" — reviewers approving without really looking. Tines' blog recommends categorizing by impact size and reversibility. Automatically allowing reads and searches while requiring approval only for irreversible operations is a realistic approach.
The talk by Gilbert also touched on operational costs. Using open-weight models (AI models available for anyone to obtain and use) in combination can cost roughly one-sixth as much as frontier models (the largest, most advanced models).
Even so, about 80% of users reportedly still choose frontier models.
The cost per token (the unit measuring the amount of text an AI reads and writes) has dropped over 90% per year, while usage volume has grown more than 50-fold — so total expenses have actually increased. Gilbert compared this to "Jevons' paradox," the phenomenon where greater efficiency leads to greater total consumption.
An example was given: compared to a top-tier model costing roughly $3 per task, a model scoring about 10 points lower on benchmarks can cost under $0.10 per task. The key question, the talk argued, is whether the entire task can be completed at the lowest possible cost.
A well-designed harness makes it easier to trust cheaper models with important tasks.
A. It's less a calculation error and more a failure to communicate unit information. The explanation is that 4,800 in the data, denominated in cents, was treated as dollars when the tool was called.
A. Not exactly. Guardrails are often discussed as a component of a harness, while harness is a broader term that encompasses logging, permission control, isolated environments, and more. That said, definitions are not standardized across the industry.
A. Too many approvals lead to rubber-stamping — things getting approved without real review. It's more practical to limit approvals to irreversible operations and those with external impact.
A. Yes. Smaller organizations tend to feel the impact of a single incident more acutely. Start with minimizing permissions and setting limits on amounts and transaction counts.
As a first step, list the permissions your AI agents currently hold and identify which operations are irreversible.
This article is a cross-post from AI Friends.
