09.14.26
In a world with little agreement, there is one thing that unites almost everyone: AI is the most transformative technology of our time. What we don’t agree on is what to do about it. Over the past few days we’ve aired a range of perspectives, from Niall Ferguson and Daniel Kokotajlo on the rising risk of AI catastrophe to Tyler Cowen on why efforts to slow AI won’t necessarily impede progress. But there’s another view of this issue—and especially the developments of the past week—that argues that the warnings from AI researchers and leaders should not be taken at face value. Shyam Sankar, chief technology officer and executive vice president at Palantir, is an influential member of this camp. In today’s Big Read, he lays out what he sees as the destructive ideology that lies behind calls for safety regulation—and why he believes what looks like a responsible plea to slow progress is actually about taking control out of the hands of the American people. —The Editors
If you turned on cable news last week, you might have seen Jacob Coxon, a software engineer who resigned from Anthropic and immediately embarked on a media tour. It all began on September 8, when Coxon warned on X that “[t]he people building AI earnestly believe that it could kill us all by the end of the decade.” A current employee at a frontier lab chimed in to confirm that, yes, he and many of his co-workers do believe there is a significant chance AI will kill us all. The odds, he said, are perhaps greater than 10 percent.
These vague and apocalyptic warnings caused a firestorm online, including calls for AI safety regulations from the leaders of almost every major AI company. But there is reason for skepticism.
The warnings from Coxon and others are infused with an ideology called effective altruism (EA). The term will be familiar to some: A few years ago, it was the idea that crypto entrepreneur Sam Bankman-Fried said motivated him. SBF, you may remember, was supposed to be the good billionaire. He was to make a fortune with his company, FTX, and give it away to protect humanity’s future, especially from big, existential risks. Instead, he stole billions from his customers and was sentenced to 25 years in prison.
Today, EA is the unseen force driving AI safety discourse. Its ideas and money course through institutions seeking to build and govern AI and, with it, America’s future. And Americans deserve to know about it, because the EA crowd is trying to restrict your access to the most important technology of the 21st century.
Let me explain.
Sam Bankman-Fried explained the EA approach to morality in a 2022 interview: “There’s something very quantitative about how you view the world if you’re thinking about things from a utilitarian perspective,” he said. “In the end, you turn things into numbers, and you decide which number is the biggest.” That approach allowed him to justify enormous fraud in the present for what he believed would produce a better future.
Effective altruism is a utilitarian philosophy that claims to use evidence and reason to do good, ranking competing needs through a common calculus. It leads in strange directions. For example, in a truly universal moral frame, effective altruists argue, why should a single country like the United States or a single species like humanity take precedence? (According to EA, shrimp welfare is an urgent priority.) Indeed, why should we value ourselves and our children more highly than the countless generations that come after us?
The movement is obsessed with quantifying existential risk—a fact that explains why discourse around AI has taken an apocalyptic turn. If tortured thought experiments can show that AI is a risk to humanity for all time, then that risk must outweigh virtually any benefit it can provide to us in the present.
The problem is, while EA philosophy claims rationality and precision, it delivers neither. And it could not be further from the views of the American people, who remain stubbornly attached to such things as their country, their children, and their freedom.
Join Bari Weiss and Dr. Simon Sebag Montefiore for a conversation on the making of the Middle East, live in New York City on October 1.
Consider what it would take to make a true accounting of existential risk. How many people might exist 100 years from now, let alone a million? What probability should we assign to a machine god destroying humanity? How much should people living today sacrifice to change that probability?