Thread

Jacob Coxon is not alone: AI Superintelligence is an Existential Threat to Humanity Recently, Jacob Coxon, a former researcher at Anthropic and OpenAI, sent shockwaves throughout the world by stating, “The people building AI earnestly believe that it could kill us all by the end of the decade.” Incredibly, in less than 48 hours, Mr. Coxon’s social media statement has been viewed by more than 150 million people. But let’s be clear: Mr. Coxon is not alone in his fears. Leading experts inside and outside the AI industry have been echoing Mr. Coxon’s clarion call for years. And that is that unless we reverse course, there is a very real possibility that once advanced AI surpasses human intelligence, it could escape our control with catastrophic consequences. In other words, advanced AI and superintelligence pose an existential threat to humanity. That is why I will soon be introducing legislation with Representative @RepCasar to pause advanced AI and ban superintelligence altogether. Please take a moment to read what just a few of these experts have said in the past few days. Thanks to @TaylorPopielarz for assembling this research. Paul Christiano, Member of the OpenAI nonprofit board’s Safety and Security Committee: "Based on the recent trajectory of capabilities and the continued difficulty of alignment, I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term. I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level … If we build superintelligence without more robust alignment I expect we will permanently lose control of it. If that happens then most people could die. I believe we would need domestic and international coordination to ensure global consistency and reduce risk to an acceptable level.” Samuel Marks, Scalable Oversight lead at Anthropic: "AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years.” Evan Hubinger, Alignment Science lead at Anthropic: "Jacob is correct here—we really do earnestly believe AI could kill all humans!” Alex Turner, former Research Scientist at Google DeepMind: "I left Google DeepMind in June. Jacob is right: many researchers believe they are building something that could kill everyone on the planet. It was literally my day job to think about how to stop that." Jakub Pachocki, Chief Scientist at OpenAI: "This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.” Jason Wolfe, Researcher at OpenAI: "I don't know what my probabilities are on literal extinction, but I think there are a number of ways AI could go poorly for humanity, and at the current frankly terrifying pace humanity will be quite lucky if we manage to find and stay on the narrow path between all the bad outcomes. Jonathan Richard Schwarz, former Senior Research Scientist at Google DeepMind: "I left DeepMind after 7 years (and since rejected offers to join the other two) due to severe concerns about the concentration of power these labs represent. What is currently happening in this field is deeply unhealthy for society.”