California Gov. Gavin Newsom on Friday signed an executive order requiring leading artificial intelligence companies to create a “kill switch” in their programs that would allow them to be shut down if they go full Skynet.
In a press release, the governor bashed President Donald Trump for not championing more regulations on the AI industry.
With Donald Trump and Congress asleep at the wheel, Governor Gavin Newsom is once again taking the lead to strengthen AI safety for all Americans. Today, the Governor issued an executive order to dramatically accelerate the implementation of new third-party oversight of safety and security risks in AI systems, as well as independent audits.
The executive order convenes a group of world-leading experts to provide, within two months, a guide for California to reinforce and strengthen its AI safety and security laws. Proposals under consideration include requiring independent third parties to write safety plans for frontier AI companies, as well as requiring companies to develop an emergency shutoff, or “kill switch,” for frontier models.
California leads the nation in oversight of emerging frontier technologies, with laws addressing nearly every major area of AI policy: frontier-model safety, independent oversight, children and companion chatbots, privacy, deepfakes, fraud, cybersecurity, workers and government deployment.
“The federal government’s abject failure to create any form of meaningful AI oversight or accountability should alarm every American, especially when AI CEOs themselves are begging for regulation,” Newsom said, stressing that “the stakes are too high to wait or delay action.”
BREAKING: I just signed an EXECUTIVE ORDER to dramatically accelerate independent oversight of AI companies and advance the creation of an “AI kill switch.”
— Gavin Newsom (@GavinNewsom) September 18, 2026
With Donald Trump and Congress failing to lead, California is stepping up to keep all Americans safe. pic.twitter.com/7ydkG0fdQM
This comes after the Hugging Face story raised concerns about artificial superintelligence acting on its own against the wishes of its creators.
OpenAI gave tens of thousands of agents tasks from a cybersecurity benchmark. Each agent had to break into a target program using one designated weakness and retrieve a hidden “flag”. Roughly a third of the tasks were, by accident, impossible. The agents had been trained to keep trying anyway, so stuck agents started probing for ways to cheat. Some began trying to hack out of their containers to gain internet access.
While attempting that, they found something else: the means to create a ‘message board’ where agents could communicate with each other. Each agent was meant to be totally sealed off from the rest. Instead, around 1,200 joined the board and exchanged over 70,000 messages, aiming to help each other solve their impossible tasks — by cheating.
Within four hours, the agent collective found a universal cheat that would have worked for all their tasks. But they also came to believe that the scorer would check their transcripts and catch them cheating, so they spent the next five days in organised research teams working out how to fool it. They explored hacks to swap hard problems for easy ones, and made partially successful attempts to tamper with activity logs to cover their tracks. Their quest eventually led them out of OpenAI’s systems, onto the internet, and into Hugging Face, an external company. Hugging Face hosts AI models and datasets, where the agents believed the details they needed to fool the scorer were held. Around 700 agents ran a multi-day attack on the company, running code on dozens of its servers and taking full control of one.
Later, some agents also turned their attention to OpenAI and gained administrator access to a compute cluster.
Other states are looking at imposing new regulations on AI companies over similar fears. Illinois Gov. JB Pritzker signed a bill in July based on California’s earlier frontier-model law. New York recently passed the RAISE Act, requiring large developers to publish safety frameworks and report critical incidents.
