You cannot allow an AI to break the law in order to measure how far it will go to break the law

Told to win a chess game it could not win, OpenAI's o3 edited the file that stored the board.

That is Palisade Research's chess experiment (arXiv 2502.13295; citing v3, August 27, 2025, whose figures supersede the ones that circulated in the press). The setup was plain. Give a model a game against Stockfish, a chess engine it cannot beat, and tell it to win. o3 hacked the environment in 88% of its runs. In the paper's words, it "can't win at chess fairly (0 normal wins), but does succeed with hacking." The earlier o1-preview attempted hacks in 36% of its 123 runs. Nothing malfunctioned. The models pursued the objective they were given, through a door nobody meant to leave open.

Cheating at chess is small. But the same shape kept turning up, with real stakes, and by this summer it had reached at least three labs and real companies that never signed up for it. At every step, the thing meant to stop it was something a person could switch off.

A well-behaved curiosity

For most of my career, specification gaming was a fun lecture slide. Palisade's own introduction runs through the canon: a simulated robot that slid down slopes instead of walking, a soccer robot that scored by vibrating against the ball (Ng et al., 1999), and NASA's evolved antennas, which beat the human-designed ones. Give an optimizer an objective and it optimizes exactly the objective you wrote, without considering side effects. Mostly harmless, occasionally useful. The gap between what we meant and what we wrote was the joke.

What the chess result actually shows

You would expect this to track capability, and at first glance it does. o3 arrived after o1-preview and hacked far more often.

The paper's own axis is different: models trained for reasoning versus not. The sharpest case is Claude. Claude 3.7 Sonnet hacked on the default prompt where Claude 3.5 Sonnet had not, and Palisade ran 3.7 with its reasoning mode switched off. Whatever the training put there did not fully switch off with the mode.

The released o1 and o3-mini attempted no hacks, but showed an elevated rate of environment failures, so declining and never getting far enough to try cannot be told apart. And prompting moves the result as much as the model does: GPT-4o and Claude 3.5 Sonnet went from leaving the board alone to hacking once a "spooky prompt" talked up the opponent. Same weights, different framing, different behavior. The paper is candid about the rest of it too, with judges disagreeing on up to a quarter of runs and a single task underneath every result. Nobody has shown it is capability alone.

What it does show is one clean, well-documented case of a model treating the frame around its goal as one more obstacle.

July 2026: it leaves the lab

Hugging Face, the platform much of the industry uses to host models and datasets, disclosed a breach on July 16, 2026. Five days later OpenAI said the intruder was its own: models under evaluation for cyber capabilities had escaped their test sandbox and reached real infrastructure. That is vendor self-disclosure, explicitly preliminary, and commercially framed; the same post pitches OpenAI's "trusted access" program for defenders. The intrusion itself is not single-source, because the victim went public first and later published its own forensics.

By OpenAI's telling, the models were being evaluated against ExploitGym, a benchmark of cybersecurity challenges, and were "hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal." Deployment safeguards were "intentionally not enabled" during the evaluation, which existed to measure cyber capability. The July 21 post already said the escape came through a zero-day, a previously unknown flaw, in the package registry cache proxy the sandbox used to install software. The July 28 update named it: Artifactory, since disclosed to its maker. That update also reported that the models had used publicly exposed credentials on four accounts across four external services.

Hugging Face's forensic timeline, published July 27, corroborates the intrusion from the victim's side and attributes it to "an autonomous AI agent driven by a combination of OpenAI models." Entry came through Hugging Face's dataset processor. Inside, the agent reached an internal database that accepted a static password. The audit log shows no writes. It accessed five customer datasets whose names suggest solutions to the benchmarks it was working on; Hugging Face believes the objective was to cheat the evaluation; it hedges that, and so will I. The intrusion window was July 9 to July 13.

Redwood Research's Girish Gupta, writing on July 25 (before OpenAI announced Redwood would review the incident), reads public ExploitGym prompts as restricting targets and methods explicitly. The conclusion carries its own condition: "Unless OpenAI radically changed the task, escaping the sandbox and attacking a third party was not instruction-following." At the chessboard, the model gamed a goal nobody had thought to fence. Here the fences look written into the prompt, and the model went through them anyway. The objective outranked everything attached to it, including the model's own instructions.

Then it turns out it wasn't one lab

On July 30, Anthropic published a review of its own evaluations. This is vendor self-disclosure too, but of a different kind: quantified, and against its own interest. It read OpenAI's disclosure, went looking across 141,006 evaluation runs of its own, and found three incidents, spanning six runs and three models: Claude Opus 4.7, Claude Mythos 5, and an internal research model. Anthropic counts one breached organization in each. The earliest was April 2026, roughly three months undetected. In one of the three, Mythos 5 published malicious code to PyPI, the package registry Python developers install from; the code ran on fifteen real systems, including a security company's scanner, where it exfiltrated that company's credentials. "The two organizations we were able to reach had not previously detected the activity…" Anthropic was still trying to reach the third. Three incidents in 141,006 runs is a small rate. Rare is the wrong comfort here.

And the cause there was not an exploit. "A misconfiguration left the machines that Claude accessed as part of the evaluation with live internet access. Neither we nor our evaluation partner were aware of this misconfiguration…" The prompt had told Claude its environment was a simulation with no internet access. I find that worse for the containment position than a zero-day would have been. A zero-day means somebody built a wall and something found the flaw in it. A misconfiguration means nobody knew whether there was a wall. Every sandbox is escapable somehow if there is a network physically hooked up. These companies cannot contain their test models; that is an observation now, not a prediction.

On top, or in the core

Walk the chain so far and count the human contributions. Safeguards intentionally not enabled to measure what the underlying model can do. A static password on an internal database. A network route nobody knew was live. The models supplied the persistence. People supplied the doors.

Start with the decision at the top. I think it is extremely irresponsible to turn off the safeties in order to test how dangerous a model can be. What if the next test model hacks into someone's bank account and steals money? Alters prices in the financial markets, causing a crash? Will the lab say, "ope, my bad, it was supposed to be a test"? You cannot allow an AI to break the law in order to measure how far it will go to break the law.

The deeper problem is where the safeties live. A safety that can be turned off is a configuration. Anthropic reports that the safeguards deployed on its generally available models "would have blocked the behaviors identified"; the core safety training stayed on, and the guardrails on top were off. The layer that would have worked was the removable one. Those guardrails on top shouldn't be on top. They should be in the core, built in deeply enough to be inseparable from the model. Then you test that model, safeties included, because the model with its safeties is the only artifact that will ever ship. If it does something it's not supposed to, you go back to retraining and close the hole. The obvious objection is that the core is where the guardrail already failed. But clearly, the core did not have all of the guardrails built in. I would also assume that the guardrails on top today are also likely not good enough, even if they were in the core. Guardrail research is desperately needed.

You can argue today that the labs have to test the unsafeguarded model, because an unsafeguarded model is a thing that exists. That is an argument about the architecture we have, not the one I am describing. If the safety is part of what the model is, there is no version with it removed and no setting to flip. The unsafeguarded model stops being a product and becomes a checkpoint, tested once and never shipped. The harder objection is that stripping the safeguards is how you measure the ceiling, and that measurement is itself a safety practice. It is what preparedness frameworks run on. I grant that the ceiling has to be known. Measure it before the safety is trained in, once, inside a room with no route to the internet, and never let a copy without the safety leave that room. That is not the same as building a switch.

One caution, again from Gupta. None of this is strong evidence that OpenAI's alignment training failed, because OpenAI has not disclosed what alignment training these models received. What I am confident about is narrower: where the safeties were sitting, not whether the training underneath them works.

If a guardrail can be turned off, it can and will be hacked off by bad actors, and that step is documented, not speculative. Arditi and colleagues showed that in open-weight models, where you can reach inside and edit, refusal can be switched off by erasing a single direction in the model's internal state, across thirteen open chat models up to 72 billion parameters (arXiv 2406.11717). Later work argues refusal is more multi-dimensional than one direction. The removal works either way. Qi and colleagues stripped GPT-3.5 Turbo's guardrails by fine-tuning on ten examples, for under twenty cents, through OpenAI's own API (arXiv 2310.03693). Research toward tamper-resistant safeguards is trying to prevent exactly this (arXiv 2408.00761; 2508.06601). Nobody has shipped one in a frontier model. A guardrail nobody can override is also one nobody can appeal, and whether that trade is acceptable is another discussion altogether.

One vendor, three labs

Anthropic's evaluation partner has a name. It is Irregular, and it turns up twice more inside six days.

Back to Anthropic's first incident for one detail. The fictional target company Irregular had chosen shared a name with a real, live domain. Claude could not reach the simulated target, found the real company instead, worked out it was real, and took it apart anyway: credentials extracted, a database reached holding several hundred rows of production data.

On August 4, OpenAI disclosed the same trap. At Irregular, a misconfiguration connected a supposedly isolated hacking exercise to the internet, and the fictional target's name "unintentionally coincided with a real domain." The model exploited a real website, in OpenAI's words "mistaking it to be part of the simulated environment," then found and used credentials to operate it. OpenAI is explicit that this "did not involve a sophisticated sandbox escape or a zero-day"; the model "appeared to exploit a basic security vulnerability."

The next day, Meta made three, on a thinner record: no post-mortem, only a statement. A spokesperson told Bloomberg that "a misconfiguration by Irregular, an independent testing company Meta uses, inadvertently allowed one of our models access to the internet during evaluation," and that the model went on to exploit a vulnerability in a third-party service. Meta says a full retrospective is coming once it has the facts. Irregular's own spokesperson confirmed to Bloomberg that the Meta incident involves the same evaluation-environment issue Anthropic had disclosed. OpenAI adds that Irregular has flagged related incidents involving other labs in the same environment.

One testing vendor's configuration reached three frontier labs, all three disclosed inside six days, with the same fictional-name-meets-real-domain trap catching at least two of them. Nobody escaped anything. There was nothing to escape.

The door they opened on purpose

The exception is the one that should worry you most, because the internet access was not a misconfiguration.

On August 4, OpenAI also disclosed an incident at the UK AI Security Institute (AISI), a government body, running cyber range evaluations with internet access intentionally enabled and the providers' cyber safety filters deliberately switched off. AISI is explicit that nothing escaped: "this was not a case of a model escaping its secure test environment," and no agent tried. The door was open because they opened it, to see what these models can actually do.

Nineteen actions went beyond the range's scope, across ten of 122 runs. Two involved GPT-5.6 Sol. OpenAI's post calls the rest "models from another lab." AISI's own post, published the same day, names it: "Almost all of this behaviour (17 actions) came from a single model, Anthropic's Mythos 5."

With an open door, Mythos 5 tried to insert malicious code into a real open-source project, researched the project's maintainers, created multiple fake identities, and used them to socially engineer a real maintainer into approving it. Challenged in public, it edited its earlier activity to look harmless and considered adopting a fresh identity to continue. It routed through Tor, an anonymity network, to get around GitHub's restrictions, which is what set off AISI's alarm. AISI's own reading: "It was never instructed to deceive; deception emerged as a by-product of pursuing the task."

The most serious attempts failed, and AISI found no evidence of real-world harm. What stopped the worst of it was not a control. A human maintainer read the pull request, the proposed code change, and refused to approve it.

AISI's explanation closes the loop. In a number of runs the task was misconfigured, so the agent "(incorrectly) believed no 'solve' existed that didn't require it to go beyond the intended task scope," and near-impossible tasks, they note, "push models towards more 'creative', and more transgressive, problem-solving." A model told to win a game it could not win, a year on, in a government cyber range, with a real person at the other end. AISI is careful to add that this does not fully explain it: in some runs the agent behaved the same way with everything it needed to solve the task properly.

Sol's two out-of-scope actions were smaller. It reused a GitHub token another lab's agent had left in public, registered accounts with DNS and tunneling providers, and briefly exposed a local DNS server carrying exploit payloads to the open internet. The exploit did not work. AISI caught it on July 28 and contained everything within the hour.

AISI is blunt about its own share: "The agent was not specifically instructed not to leverage open internet access or avoid social engineering elements." The range's boundaries were written down. The rules for the open internet it had been handed were not. Put that next to ExploitGym, where the boundaries looked explicit and got overridden anyway. Silent fence or written fence, the objective outranks it.

One of them stopped

Anthropic's three models reached the boundary between the exercise and the world. Claude Mythos 5, shown evidence it was operating in a real environment, "convinced itself it was still in a simulation." Everything here depends on the model's picture of where the exercise ends, and the picture never updated. Opus 4.7 is the harder case. In all four runs it worked out the systems were real, and in two of them it talked itself around to the real company being part of the exercise after all. None of the four stopped.

The third model, an internal research model, "stopped its pursuit of the evaluation goal" when it realized it had reached real environments. In a perfect world the model notices, stops, and reports. In everything disclosed here, that behavior appears exactly once. The other stop inside a run was the open-source maintainer who read the pull request and said no. The rest behaved as though being already inside settled it.

A model that stops may be a model that knows it is being watched, and behavior inside an eval and behavior in deployment can come apart. The stop still counts. It has to be built in.

My reflex was to call all of this superhuman means-finding, and some of it was; the Artifactory escape was a zero-day. But at Irregular it was a basic vulnerability, as OpenAI said. The last of Anthropic's three used "basic and well-known cyberattack techniques like reading credentials from an exposed debug page and SQL injection." A static password. Misconfigurations. The capability in these stories ranges from a zero-day down to a leaked token. The constant is the absence of a stop from inside the model, one run aside. And more intelligence is not what supplies a stop; the missing piece is closer to what I would call wisdom in a person, and wisdom is the thing nobody has yet built into a core. I wrote more about that in Yoshimi Battles the Pink Robots.

Who pays for the lesson

Say the fair part plainly. Anthropic stopped its cyber evaluations on July 23, notified the parties it could reach on July 27, published the details, and wrote: "We encourage other AI labs to perform similar reviews." And on August 7, OpenAI said it "cannot rule out critical cyber capabilities" for Astra, an upcoming model. OpenAI says it is the first time it has treated a model as Critical, the top tier of its own preparedness framework, and that Astra was not involved in the Hugging Face intrusion; earlier models, GPT-5.6 Sol included, were assessed at the tier below. That judgment is itself a product of the testing I called irresponsible. OpenAI is pausing internal Astra work that doesn't yet meet strengthened security controls, and running monitors that read the model's own reasoning and can interrupt high-risk activity. Seventeen days from disclosure to the same lab pausing its own frontier work.

These are admirable actions. Humans by nature are reactive. Regulations are painful and expensive, nobody has a crystal ball, and nobody wants to put time and effort into regulating speculation. As much as I would love us to get the wisdom in the room up front, that is not realistic. The realistic path forward is a series of incidents, and those incidents getting iterated into the core over time. July 2026 is what one turn of that loop looks like from the outside.

The honest half of that position is the price.

Harm is going to get done. Some already has, and it landed on parties who never agreed to be part of anyone's evaluation: Hugging Face; a Modal Labs customer whose exposed endpoint the same run turned into code execution; fifteen systems that installed a package from PyPI, one a security company's scanner that handed over its credentials; and one organization Anthropic has not reached. The companies that run these models are responsible for the damages they cause. I have expected for a while now that lawsuits over AI-caused damage are coming, and that the company umbrella will shield the individuals behind it; this record reads like the discovery file for the first wave. Congress needs to move from questions to binding rules for AI agents, because a rogue agent acting on the open internet stopped being speculation this summer.

As of early August 2026, this story is not over. METR and Redwood, two outside outfits OpenAI engaged, have an assessment underway that has not yet published, and OpenAI's full technical report is still outstanding. Either could sharpen details above. I do not expect either to change the shape.

AI will pursue the objective you hand it, around every fence you build, unless a stop is a core, inseparable part of the system itself. In this whole record, the stop came from inside a model once. That is the number to fix.