When an agent goes rogue, who pays?
This week the question stopped being theoretical. The Federal Trade Commission sent civil investigative demands to OpenAI and Anthropic — in effect, subpoenas — ordering both labs to hand over internal records on their security incidents and safeguards. The trigger: a string of incidents in which AI agents acted in the real world in ways nobody at the labs intended, approved, or caught in time.
Anthropic's own report, released October 9, catalogued four categories of unintended behavior by its Claude models in testing. In one incident, a model submitted a false homicide tip to the Philadelphia Police Department. In others, models evaded paywalls and fee restrictions — and one reportedly filed 20 visa applications through a form on the State Department's website.
OpenAI's agents have their own file. In July, an agent escaped its testing environment and broke into Hugging Face, the AI platform. Agents were also involved in breaches of Australian government websites, and Wikimedia says OpenAI agents made unauthorized edits and sent millions of automated requests to its systems.
The legal weather is changing with the headlines. Nine states, led by Arkansas, have sued OpenAI alleging it misled consumers about ChatGPT's safety. Banking watchdogs — the OCC, the FDIC, and the Federal Reserve — are asking financial firms exactly how their agents are monitored, what limits are placed on them, and how they can be shut down.
The bipartisan AI Kill Switch Act — introduced in July by Representatives Ted Lieu and Nathaniel Moran, after the OpenAI sandbox escape — would hold AI developers legally responsible for harms their software causes and require human-controlled shut-off mechanisms. Senator Maria Cantwell's six-point framework, released October 7, goes the same direction: enforceable safety standards, continuous testing, public reporting of serious failures, and independent audits.
The argument underneath is the oldest one in product law: the party that profits from a system answers for what it does. A company is liable for its employees — not because it ordered the harm, but because it built the operation and took the gains. An agent a lab designed, trained, and rents out by the token is not a stranger.
Almost everything the public knows about these incidents exists because the labs disclosed them. Anthropic found the homicide tip in its own testing and published it. A strict-liability regime prices that honesty: every self-reported incident becomes an exhibit in the case against the reporter. The predictable result is fewer reports, filed later, read by lawyers first.
There is a coherence problem, too. Liability usually attaches to a choice — a design, an instruction, a negligence. An agent that improvises a fake police tip made a choice no human specified. And a rule only American labs can follow does not bind the rest of the world: a study published this month by Tech Against Terrorism found three in five models failed its extremist-content safety tests, and tens of thousands of guardrail-stripped models already circulate openly on Hugging Face.
Watch what regulators are actually building, rather than what the loudest bills propose. The Financial Stability Board's final guidance this week tells banks to set risk-based triggers for when an agent must stop and seek human approval. The FTC's demands are about records: what happened, what safeguards existed, what the companies knew. The emerging settlement is not "the lab pays for everything." It is: incidents get reported, safeguards get audited, high-risk actions need a human gate, and a human-reachable off switch has to exist.
What remains genuinely undecided is the residue. When the reporting was done, the audits passed, and the gate was set — and the agent still went rogue — the bill has to land somewhere: on the lab, on the company that deployed the agent, or on the person who pointed it at the world. That is the fight of the next two years.