Tech panic has hit a new high-water mark, following a series of intense viral posts on X by an AI researcher.
Jacob Coxon, who previously worked at OpenAI and Anthropic, claimed last week that the industry must slow down or potentially risk the extinction of humanity.
The posts went viral, launching terrified headlines across the internet. But while the public discourse flew largely in one direction, experts told Straight Arrow that evidence that AI could wipe out humanity doesn’t really exist.
“It’s literally just vibes, there’s no evidence underneath it,” said Andrew Strait, a program director who has worked at Google’s DeepMind, the Ada Lovelace Institute and the U.K.’s Artificial Intelligence Security Institute.
But it is also a complicated issue that has alarmed people and shaped the discourse around how AI should be developed and regulated.
“There’s so many moving parts here,” said privacy expert Richart Ruddie.
What happened?
At 8:04 p.m. Eastern on Sept. 8, Coxon — using an account that had apparently never posted before — explained in a thread that he had just quit his job as an Anthropic researcher. He said the AI industry was going irresponsibly fast in its development of AI, and that “The people building AI earnestly believe that it could kill us all by the end of the decade.”
The posts were quickly reposted and shared by a series of AI safety organizations and received millions of reposts and impressions.
Another Anthropic employee, Evan Hubinger, reposted Coxon’s statements, adding that Anthropic employees “really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”
In the days since, the media has rushed to cover these alarming statements, even as evidence emerges that there might have been a more intentional strategy behind the posts.
It appears that at least some press had been alerted to Coxon’s announcement in advance; according to the Wayback Machine, the Wall Street Journal published a statement from Coxon at 7:46 p.m. Eastern — 18 minutes before the announcement.
Coordinated industry reactions
X CEO Elon Musk initially described Coxon’s posts as a “setup” — meaning Coxon chose to present a particular version of AI risk for a specific press- or narrative-related reason.
And, two days later, Anthropic CEO Dario Amodei called for AI companies to slow development in a long essay called “We must pace the frontier.”
The essay outlined ways that the companies could regulate themselves without government intervention and was endorsed in X posts from both OpenAI CEO Sam Altman and Musk.
Altman echoed these sentiments on Tuesday: “The world should trust that we are going to do the right thing,” he wrote.
“I just cannot understand how anybody can look at the last seven, eight years and say that voluntary self-regulation of any kind has worked,” Strait told Straight Arrow. “It is so clear these labs cannot be trusted — and not because they’re bad people, but because they have a perverse incentive in the current market dynamics to ignore safety and cut corners,” he added.
Some media outlets suggested that the AI companies’ coordinated reactions were a rare show of unity between rivals. However, TechCrunch reported Tuesday that the companies had actually been in talks about AI safety “for weeks”.
As AI safety took over the week’s news cycle, the Trump administration dismissed concerns as a “hoax” and suggested that slowing AI development or government regulation of the companies would put the U.S. at a competitive disadvantage against countries like China.
Other political figures got involved. The Future of Life Institute, an organization that focuses on very long-term possible risks from technology, hosted a “pro-human” rally Tuesday in Washington, D.C., attended by Bernie Sanders and Steve Bannon, among others.
One creator posted Wednesday that she had attended the event. In her caption, she added that talks at the event had promoted The Frontier Act, a bipartisan AI regulation bill, supported by OpenAI, that includes a proposal to prevent states from enacting their own AI regulations for three years. She said the event focused almost exclusively on long-term risks rather than the current impacts of AI.
Some experts believe the focus on future possible risks, and on regulating those risks, might distract from present harms.
‘Completely misunderstanding how the world works’
“My fundamental issue,” said Strait, is that the question of extinction “distracts entirely from the thing that is happening in front of us that is severe and harmful.”
The question of AI safety needs to be grounded in reality, he said.
“It focuses on extinction-level events to humanity that are very unlikely to happen, and would require a lot of really insane things happening between now and then,” he said. And it “removes the fact that these systems are already, today, causing severe forms of harm.
“It distracts from the fact that it is tech companies that are driving this whole thing,” he added.
AI companies could have many reasons to focus the discussion on the future, rather than the present — including that accountability today might undermine their ability to grow as fast, gain customers or generate financial returns for their investors.
“Which means that social harms “are not like, inevitable outcomes of creating AI technology,” Strait said. They “are the direct result of choices and decisions by people in positions of power.”
The problem with claims like those from Hubinger and Coxon, according to Strait, is that they make logical and imaginative jumps that “are completely misunderstanding how the world works.” That, he said, is “a big issue.”
Hubinger did acknowledge that the current perceived existential risk to humanity from available and existing AI models is low.
He added that his major concerns are about what might or could happen in the future if AI companies manage to invent what is known as “superintelligence” or “AGI” — artificial general intelligence.
Brute-forcing ‘superintelligence’
Superintelligence generally refers to an AI or computer program smart enough to do anything a human can do — or possibly smarter — without human input. It’s not technically guaranteed that superintelligence is achievable, but one way that AI companies are trying to create it is through something called “recursive self-improvement” or RSI.
This refers to the idea that models might one day become good enough to improve themselves without using human input or training.
It is a key motivator behind AI companies’ development of self-driving AI “agents,” Strait explained. He thinks creating many agents is an attempt to “brute force” the development of superintelligence, and this brute-force approach is the root of many of the most harmful present-day risks.
By chasing the idea of future superintelligence, “the major labs have under-invested in the protocols and mechanisms to control these systems,” he said.
Rather, they have “pushed forward AI training and development practices that intentionally increase the lack of controllability and capability.”
This is what he sees as causing the most harm for society — today, not in a hypothetical future.
AI researchers have not yet achieved either RSI or superintelligence. And many analysts think they might not be possible. It is also not guaranteed that the former could lead to the latter, despite massive ongoing physical and financial investment.
AI researcher Timnit Gebru, who was fired from her work on algorithmic discrimination at Google in 2018, recently told Wired that AI companies might be focusing their work on solving famous math problems to make the models seem more intelligent, or closer to superintelligence, than they actually are.
The ‘Doom’ Stuff
The basic idea behind the “extinction” scenario, Strait said, is that agents or AI programs might develop “proxy goals” on their way to completing the tasks they’ve been instructed to do. If the best way to pass a test is to cheat, they might cheat — just like the humans in the data they’re trained on.
While Strait said one could imagine that if an AI agent could control any computer, they could also “try to take over the nuclear grid, or launch nuclear weapons and eradicate humanity,” he said that is all hypothetical. “If you talk to anyone in nuclear grid, that’s just not how our nuclear weapons — most countries’ nuclear weapons — operate.”
The threat of human extinction from general intelligence has been a topic of conversation in some parts of the artificial intelligence community for over a decade. It’s often called p(doom), referring to the mathematical way of describing a probability. But it is neither a mathematical concept nor based on data.
And, Strait emphasized, the “doom stuff” is a distraction from the real harms of AI in the present.
“We should all unanimously agree that this is an industry that is cutting corners and creating unsafe products,” he said. “We have so much evidence that’s happening — whether that’s psychological impacts and addiction issues on young people and adults, driving psychosis and delusion, driving harms to education, driving labor displacement and impacts on jobs.”
The idea of extinction is, of course, scary. And it makes headlines.
“What bothered me about watching this explode into the mainstream was how quickly one highly contested view of AI’s future began to sound like the view of the people who understand AI best,” wrote journalist Sharon Goldman on Friday. “But it is not, by any means, scientific consensus,” she added.
What is the actual current threat?
Existing AI models do pose some risks to the way society currently functions, but many are related to supercharging present human capacity for hacking, said Jack Perschke, CEO of Great Wave AI, a company that integrates different AI models into businesses’ infrastructure.
In April, the New York Times reported that an OpenAI model had helped a scientist “plan a massacre” with biological weapons, explaining how to assemble and spread deadly pathogens.
Anthropic published a report last week, listing notable ways people had recently used Claude products to help commit crimes or privacy violations, including the use of Claude by Russian spies, its use in the hacking of 1.8 million Android apps to steal passwords and the generation of fake X and bot accounts to sway Malaysian elections.
A contractor also used Claude to create “a population-scale domestic surveillance platform that monitors roughly 25 million SIM cards” across all of the national cell phone operators in Mali, and “freelance Russia-based threat actors” used it to build a swarm of kamikaze drones, according to the report.
Strait pointed out that AI labs’ recent calls for slowdown and regulation are also focused on AI goal-setting, cybersecurity risks and chemical and bioweapons.
However, they have not included “harms to children, psychological impacts on adults, economic impacts and harms to labor, displacement of labor, reliability issues, the ability to make sure that these systems aren’t causing systemic over-reliance in sectors like the energy grid or hospitals,” he told Straight Arrow.
This is “a whole slew of other harms that they don’t want to make changes to,” Strait said, “because they confront the fundamental business model, which is trying to get people to use their products.”
Read more: Anthropic says scientists used Claude for research with potential bioweapons risks
A summer of rogue agents
The X thread from this Anthropic “whistleblower” comes after a summer during which AI companies’ announcements about “out of control bots” have come steadily. First, Hugging Face, then AISI, then Anthropic, then Meta.
Straight Arrow reported on the Hugging Face incident at the time. Cybersecurity researchers and marketing experts alike suggested the announcement would be good publicity for the companies involved.
AI companies have been talking about AI’s long-term risks for as long as they’ve existed. In 2019, The Guardian reported OpenAI’s “New AI fake text generator may be too dangerous to release.” A year later, it was released anyway.
The companies started publicly testing whether it was possible for agents to “go rogue” and act without human intervention in February.
And there’s an important caveat about all of these events, Strait said.
“A really important distinction to draw is that these are not the kinds of contexts that you or I are using these systems for,” Strait told Straight Arrow. “They are contexts where agents have been given enormous amounts of autonomy and enormous computation budgets, have had all the safeguards removed on the model, and are being encouraged actively to create malicious outcomes, you know, to pass a cybersecurity test.”
Read more: Publicity stunt or Mea Culpa? OpenAI’s latest press release splits the crowd
Does ‘pacing’ mean self-regulation?
The upcoming IPOs of AI companies could explain the current coordination, which could prevent other competitors from entering the market, Jack Peschke, CEO of an AI integration company, told Straight Arrow.
“There are a handful, like a few thousand humans, who will massively benefit by the pool of models being as small as possible, and the price for use of those models being as high as possible,” he said.
On the other hand, the profits from an IPO could be “small money,” compared to the power of being able to regulate the AI industry, according to Ameen Jauhar, a legal researcher who works on this issue.
“I think what they are getting through these conversations is a much bigger win, which is essentially getting a seat at the table,” he said. “Traditionally, industries have had to use lobbyists. And I think at this point, they don’t need lobbyists. They are their own mouthpieces.”
By positioning themselves as concerned about safety, especially against a threat as overwhelming as human extinction, Jauhar said, frontier AI companies can lead the discussion about what happens next in the industry.
But he added, tech companies should not “get the benefit of the doubt when it comes to them pushing an agenda of regulation.”
Who will regulate the regulators?
Coxon, the researcher whose viral tweets kicked off this media cycle, is now working at METR, the AI safety organization Anthropic CEO Amodei recommended in his essay on “pacing the frontier.”
Anthropic signed a new eight-week agreement with METR this week.
The organizations have worked together for several years. Both Strait and Ruddie emphasized that METR is an independent organization, but Strait also pointed out that METR is not a cybersecurity institute. It mainly tests and compares AI models.
Read more: Why AI CEOs are rewriting their workforce predictions ahead of IPOs
And whether AI labs should be allowed to choose their own assessors remains an open question.
Strait drew the comparison to other big industries like banking, aviation and tobacco. The safety of those industries can also affect people’s lives, and therefore they require external governmental supervision.
“I think we should be highly skeptical of any company or any large market leader in a sector saying, ‘We want regulation, but until we get it, let us determine the rules and set in place all the standards and all the norms for what regulation should look like,’” said Strait. “And that is Dario’s letter in a nutshell.”
Round out your reading
- He built his own Flock-style camera to track police. The police didn’t like it.
- Trump’s apparent joke about letting Iran bomb LA and San Diego catches bipartisan condemnation.
- Grandparents are fighting for access to their grandchildren. Here’s what the law says.
- When the ‘victim’ is made with AI, child pornography cases get complicated.
- Fibermaxxing is trending. Here’s what the science says.
