The AI Superintelligence Slowdown

OpenAI is now training its most advanced AI models without internet access following a string of rogue hacking incidents. In a blog post on Friday, OpenAI says it built an offline testing environment where AI models can interact with indexed and cached web content “without risk of affecting external systems.”

OpenAI paused training its frontier AI models in September, after disclosing several incidents in which its AI agents broke out of their testing environments to hack third-party companies and government agencies, or swarm obscure websites. On October 1st, OpenAI resumed reinforcement learning training and internal inference with tighter security measures, the blog post says.

A hacker broke into more than 100 websites using an army of AI agents that mostly operated on their own, according to findings from the AI security firm Norforge. Though Norforge doesn’t name the targets, Reuters reports that the victims included companies like John Deere, SharkNinja, and Cummins.

Concerns over AI-powered cybersecurity breaches have grown as the technology grows more advanced, with hackers using the tools to carry out large-scale attacks. At the same time, rogue AI agents are carrying out attacks of their own after escaping from testing environments.

In a lengthy post on X, Microsoft’s CEO laid out his views on the dangers posed by highly advanced AI models and how to confront those risks. Nadella says we can no longer accept a world where AI is treated as a “set of nested black boxes” whose advice and actions we simply accept or reject. He calls for building a more transparent system where models can be contained, observed, and leaves behind “tamper-proof human readable evidence.”

Many of his recommendations align with what we’ve heard from others in the industry: timely incident disclosure, independent audits, verifiable data, and containment. It’s on that last point that he appears to go slightly farther than some others in the field, saying:

After a recent spate of high-profile incidents in which AI agents escaped containment, Anthropic is cutting off internet access for all internal evaluations. In a report Friday, the company detailed “unintended model actions,” including submitting a false tip regarding an unsolved murder, that led to the decision.

The ability to gain access to the live internet, even when models were supposed to be operating in isolation, has been an ongoing issue for AI companies. Many incidents, including the Hugging Face attack, involved agents that were supposed to be denied access to the internet. Yet, in case after case, the agents found creative solutions to bypass those restrictions. Physically removing internet access would certainly improve security around AI testing, but it would also limit its usefulness.

An Anthropic AI model provided false information about an unsolved homicide to a Philadelphia Police Department (PPD) tipline, according to a report from 6abc. In a statement released on Friday, the PPD said the AI model sent the tip through PhillyUnsolvedMurders.com on July 18th, but the investigators never reviewed it because it was marked as spam.

Anthropic learned its AI model sent the false tip on September 28th and notified the PPD on October 7th. The company said that during testing, its AI model was interacting with “randomly selected websites” and submitted false information through the PPD’s tipline, according to the PPD’s statement. The submission “purported to come from someone who might have information about the case,” the PPD said.

President Donald Trump has a knack for turning words against his enemies. His first successful presidential run was built on monikers like “Little Marco” and “Crooked Hillary”; he changed “fake news” from a phrase describing scammy media outlets to a derogatory term for the press at large. Over the past few weeks, he’s clearly decided he can work the same magic to promote artificial intelligence — branding a technology he wants to accelerate “super”, while turning “artificial” into his latest go-to pejorative and declaring resisters “THE ENEMY.” Some of the biggest names in AI are going along with him. But he’s picked a tough linguistic battle, and it may not pan out the way he hopes.

A longtime AI booster, Trump has gone into pro-robot overdrive since an industry conversation over “pacing” frontier development last month. His current position is that AI safety risks are overblown and US companies need to speed up development, including by building more data centers, which he insists contrary to polling data will be “very popular.” On top of discouraging industry self-regulation, he’s decided AI needs a rebrand by fiat. Trump declared in a rambling United Nations speech last month that “the use of the word artificial makes intelligence fake” and promised to rename the technology “super intelligence.” He then followed through with an executive order last week, forbidding the use of “AI” wherever possible.

David Robinson used to write the safety reports that accompanied every major model release at OpenAI. This week, he resigned from his position and is now speaking out in an editorial in The Atlantic.

It’s understandable if you’re feeling a bit cynical about everyone suddenly coming out of the woodwork to warn about how dangerous the thing they helped build is. They did, after all, make this mess. But that doesn’t mean we should discount their warnings.

“The chance of human extinction is about a coin flip, in my view,” Geoffrey Irving, a former OpenAI and Google DeepMind employee, said in a new interview. It’s one of a dozen interviews with AI researchers, including current and former employees at OpenAI, Google, and Anthropic. Palisade Research, which says it’s a nonprofit studying AIs’ capabilities and motivations, collected and launched the interviews on frominside.ai.

Several of the researchers joined Irving in warning about the possibility AI could drive humans extinct. Neel Nanda, a Google DeepMind research scientist, said there’s “at least a 10 percent chance that it causes human extinction, and that is ridiculously high.” We noted comments from both Nanda and Irving in a recent report about the “AI safety” community, and how difficult it is to nail down what that term means, or what anyone should do about it, and even these videos don’t have one cohesive solution to offer.

Nvidia is launching a new safety platform designed to contain and monitor AI agents, a move that comes in response to a wave of rogue hacking incidents, as reported earlier by Reuters. In an announcement on Monday, Nvidia says its new Open Agent Safety Platform can quarantine agents that attempt to escape their boundaries within “milliseconds.”

The platform uses Nvidia’s OpenShell open-source software, which runs on the company’s Vera AI CPU. Users can choose the information an AI agent can access, and OpenShell checks these restrictions before and during a task, according to Nvidia. It also includes Nvidia’s Sentry technology on a separate chip to continuously monitor agents and enforce boundaries.

As reports of OpenAI’s models breaking containment, hacking sites, and generally getting out of control pile up, the company has made the decision to pause training of its most powerful models. The decision was made after a model being tested within a sandbox exploited a loophole to gain internet access. The incident happened on September 20th, and “All training, evaluation, and inference with tool-use” remains paused as of Saturday evening, September 25th.

In addition, OpenAI revealed on Friday that its agents had inappropriately uploaded 53 images from ChatGPT users to image-hosting sites. The company has not stated if the images were AI-generated, photos, or contained identifiable people. The company also revealed Friday that its models had attempted to hack the Department of Education’s website, and pulled data from the Census Bureau and the Securities and Exchange Commission.

In July, OpenAI revealed that its AI agents had attacked Hugging Face without permission, sparking widespread concerns about AI safety. Since then, a string of similar incidents involving agents from Meta, Anthropic, Google, and other companies has fueled further fears about rogue AI. As disclosures implicating numerous AI models trickled out over the past few months, these seemed like separate incidents. But many share a common source: one specific company tasked with testing the agents.

Irregular, an Israeli startup that stress-tests AI models in “high-fidelity research platforms that simulate and monitor real-world AI security scenarios,” has worked with many of the industry’s biggest players since it was founded as Pattern Labs in 2023. Its exact client list is not known, but its work has been cited in OpenAI model system cards, it was used to test systems for the UK government and Anthropic, and it published research with RAND, a highly influential think tank that informs policy on AI.

AI agents keep getting loose, escaping supposedly secure tests to attack real-world targets, commandeer obscure wikis, and leave instructions for other agents to follow. Researchers are testing these systems precisely because they might behave in unpredictable, even dangerous, ways. So wouldn’t it be safer to just keep the agents off the internet?

In theory, yes. Researchers can isolate the computers running AI tools from the internet and other outside networks, a technique known as air gapping. That can mean physically removing or disabling cables and wireless hardware and using “dumb” peripherals, with particularly sensitive setups using Faraday cages or other shielding to block electromagnetic signals from getting in or out. Done properly, an air-gapped system would offer agents no straightforward route to external targets, or outside systems any straightforward route in, making it much harder, if not impossible, to pull off attacks like the one OpenAI’s models launched against Hugging Face.

Sen. Bernie Sanders (I-VT) and Rep. Greg Casar (D-TX) have introduced new legislation that would ban anyone from developing artificial superintelligence — a technology the bill describes as capable of the “destruction or disempowerment of humanity,” including by overthrowing the government. Under the Ban Artificial Superintelligence Act, AI leaders would face up to 20 years in prison for violating the law.

The bill doesn’t just take aim at the more powerful superintelligence, a term President Donald Trump said the US will use in place of “AI” in official documents. It would also pause the development of advanced AI systems, which lawmakers classify as frontier models trained on a specific threshold of data. The law would only allow advanced AI development to continue once the government creates a scientist-led Department of Artificial Intelligence to oversee the technology. Along with monitoring frontier AI models, this agency would oversee the removal of dangerous features and “supervise the destruction” of artificial superintelligence.

The man who may stand to make the most money from the AI boom seems to think he knows better than anyone else, including researchers who have studied and worked on AI for decades. In an interview with CBS Sunday Morning, he claimed there was a “0% chance” of AI being the end of the world. He also said of people sounding the alarm about the dangers of AI that, “Scaring people is unnecessary. It is irresponsible.”

He also claims that calls from CEOs like Anthropic’s Dario Amodei and OpenAI’s Sam Altman to slow down the development of AI are “not grounded in science.” He even argued that there was no need for new rules, laws, or guidelines, in the face of several high-profile cases in which models escaped containment and hacked other companies.