If Anyone Builds It, Everyone Dies: How Artificial Superintelligence Might Wipe Out Our Entire Species with Nate Soares

If Anyone Builds It, Everyone Dies: How Artificial Superintelligence Might Wipe Out Our Entire Species with Nate Soares

Technological development has always been a double-edged sword for humanity: the printing press increased the spread of misinformation, cars disrupted the fabric of our cities, and social media has made us increasingly polarized and lonely. But it has not been since the invention of the nuclear bomb that technology has presented such a severe existential risk to humanity – until now, with the possibility of Artificial Super Intelligence (ASI) on the horizon. Were ASI to come to fruition, it would be so powerful that it would outcompete human beings in everything – from scientific discovery to strategic warfare. What might happen to our species if we reach this point of singularity, and how can we steer away from the worst outcomes?

In this episode, Nate is joined by Nate Soares, an AI safety researcher and co-author of the book If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. Together, they discuss many aspects of AI and ASI, including the dangerous unpredictability of continued ASI development, the "alignment problem," and the newest safety studies uncovering increasingly deceptive AI behavior. Soares also explores the need for global cooperation and oversight in AI development and the importance of public awareness and political action in addressing these existential risks.

How does ASI present an entirely different level of risk than the conventional artificial intelligence models that the public has already become accustomed to? Why do the leaders of the AI industry persist in their pursuits, despite acknowledging the extinction-level risks presented by continued ASI development? And will we be able to join together to create global guardrails against this shared threat, taking one small step toward a better future for humanity?

(Conversation recorded on November 11th, 2025)

About Nate Soares:

Nate Soares is the President of the Machine Intelligence Research Institute (MIRI), and plays a central role in setting MIRI's vision and strategy. Soares has been working in the field for over a decade, and is the author of a large body of technical and semi-technical writing on AI alignment, including foundational work on value learning, decision theory, and power-seeking incentives in smarter-than-human AIs. Prior to MIRI, Soares worked as an engineer at Google and Microsoft, as a research associate at the National Institute of Standards and Technology, and as a contractor for the US Department of Defense.

Show Notes and More

Watch this video episode on YouTube

Want to learn the broad overview of The Great Simplification in 30 minutes? Watch our Animated Movie.

---

Support The Institute for the Study of Energy and Our Future

Join our Substack newsletter

Join our Hylo channel and connect with other listeners

Avsnitt(344)

Meeting the Future Halfway | Frankly #4

Meeting the Future Halfway | Frankly #4

In this Frankly, Nate unpacks the choice of the podcast title "The Great Simplification", and how he thinks about responses - rather than solutions - to the challenges we face in the decade ahead of us. He lays out the framework for the scale and degrees for how we can elevate the chances for a positive future. He also reflects about what he's learned while hosting The Great Simplification and where he hopes to move forward in the future for the podcast. For Transcript visit: https://www.thegreatsimplification.com/frankly-original/frankly-04-meeting-the-future-halfway To Watch on Youtube: https://www.youtube.com/watch?v=8Qs_cBThk3U

2 Juli 202221min

Gerardo Ceballos: "Will the Ongoing Population Extinctions Lead to a 6th Mass Extinction?"

Gerardo Ceballos: "Will the Ongoing Population Extinctions Lead to a 6th Mass Extinction?"

On this episode, we meet with ecologist and conservationist Dr. Gerardo Ceballos. Ceballos discusses animal populations, the sixth mass extinction, his new project, Creatures United, and how we can better care about and protect Earth's remaining biodiversity. About Gerardo Ceballos: Dr. Gerardo Ceballos is an ecologist and conservationist very well-known for his theoretical and empirical work on animal ecology and conservation. He is particularly recognized by his influential work on global patterns of distribution of diversity, endemism, and extinction risk in vertebrates. Ceballos was the first scientist to publish the distribution of a complete group of organisms (mammals). He is also well – known for his contribution to understanding the magnitude and impacts of the sixth mass extinction; he has shown that vertebrate species that became extinct in the last century would have taken more than 10 thousand years under the "normal" extinction rate. For Show Notes and Transcript visit: https://www.thegreatsimplification.com/episode/25-gerardo-ceballos

29 Juni 20221h 37min

Jason Bradford: "A Hybrid Path to the Future of Farming"

Jason Bradford: "A Hybrid Path to the Future of Farming"

On this episode, Jason Bradford, who is an author, activist, farmer, and teacher, talks about the energy intensity of our modern industrial agriculture system. How do we feed billions of people with depleting energy systems? How do we also protect existing biodiversity and ecosystem health? We also discuss what makes for healthy soil, why we're losing it, and how small farms can help get it back - while creating higher yields of healthier foods for fewer inputs. About Jason Bradford: Jason Bradford has been affiliated with Post Carbon Institute since 2004, first as a Fellow and then as Board President. He grew up in the Bay Area of California and graduated from U.C. Davis with a B.S. in biology before earning his doctorate from Washington University in St. Louis, where he also taught ecology for a few years. After graduate school he worked for the Center for Conservation and Sustainable Development at the Missouri Botanical Garden, was a Visiting Scholar at U.C. Davis, and during that period co-founded the Andes Biodiversity and Ecosystem Research Group (ABERG). He decided to shift from academia to learn more about and practice sustainable agriculture, and in the process, completed six months of training with Ecology Action (aka GrowBiointensive) in Willits, California, and then founded Brookside School Farm. For Show Notes and Transcript visit: https://www.thegreatsimplification.com/episode/24-jason-bradford

22 Juni 20221h 13min

Energy Blindness | Frankly #3

Energy Blindness | Frankly #3

Nate explains how our culture is "energy blind" and the implications. The YouTube video, featuring charts and graphs, of this podcast is available now: https://www.youtube.com/watch?v=mVjhb8Nu1Sk For Show Notes and Transcript visit: https://www.thegreatsimplification.com/frankly-original/frankly-03-energy-blindness

21 Juni 202224min

Tim Watkins: "From Living Like Gods to Living Your Own Story"

Tim Watkins: "From Living Like Gods to Living Your Own Story"

On this episode, we meet with author, social scientist, policy researcher, and mental health advocate Tim Watkins. Watkins gives us a bird's eye view of how energy, the economy, the environment, and mental health fit together. How important will mental health be to help us navigate uncertain times? About Tim Watkins: Tim Watkins is the author Consciousness of Sheep, social scientist, policy researcher, and mental health advocate. Watkins has authored a range of books, including numerous books and booklets on the subject mental health, wellbeing and self-help. In 2015 he published "Austerity - Will Kill the Economy", a critique of the economic policies adopted in the UK since 2010; and "Britain's Coming Energy Crisis - Peak Oil and the End of the World as we Know it", a guide to the UK's particular vulnerabilities in a world without cheap oil. Tim Watkins is a founder-director of Waye Forward Ltd. A qualified Life Coach, he also provides coaching, mentoring and support to other writers. For Show Notes and Transcript visit: https://www.thegreatsimplification.com/episode/23-tim-watkins

15 Juni 20221h 21min

Aza Raskin: "AI, The Shape of Language, and Earth's Species"

Aza Raskin: "AI, The Shape of Language, and Earth's Species"

On this episode, we meet with cofounder of the Earth Species Project, cofounder of the Center for Humane Technology, and cohost of the podcast Your Undivided Attention, Aza Raskin. Raskin gives us a general overview of what artificial intelligence is, how it's about to become more deeply embedded in our lives, and how he and his team plan to use AI as a Rosetta Stone to translate the languages of other species to - hopefully - expand human consciousness, empathy, and awareness of the other beings we share this planet with. About Aza Raskin: Aza is the cofounder of Earth Species Project, an open-source collaborative nonprofit dedicated to decoding animal communication. He is also the cofounder of the Center for Humane Technology and is the cohost for the podcast Your Undivided Attention. Trained as a mathematician and dark matter physicist, he has taken three companies from founding to acquisition, a co-chairing member of the World Economic Forum's Global AI Counsel, helped found Mozilla Labs, in addition to being named FastCompany's Master of Design, and listed on Forbes and Inc Magazines 30-under-30. For Show Notes and Transcript visit: https://www.thegreatsimplification.com/episode/22-aza-raskin

8 Juni 20221h 49min

Vicki Robin "Money and LIfe's Energy"

Vicki Robin "Money and LIfe's Energy"

Show Summary:  On this episode, we meet with social innovator, writer, and speaker, Vicki Robin. Robin unpacks how the machine of community begins. How does being vulnerable, sharing, and being obligated to others create a system that allows everyone to contribute?  Why do we need to learn to begin asking for help? Further, Robin shares how we can begin to take steps toward food resiliency. Robin shares the story of how she only ate food that was produced within a 10-mile radius of her home for 30 days, and how we should all begin to think and act locally. About Vicki Robin: Vicki Robin is a prolific social innovator, writer and speaker. She is coauthor with Joe Dominguez of the international best-seller, Your Money or Your Life: Transforming Your Relationship With Money and Achieving Financial Independence (Viking Penguin, 1992, 1998, 2008, 2018). It was an instant NY Times best seller in 1992 and steadily appeared on the Business Week Best Seller list from 1992-1997. It is available now in twelve languages. Blessing the Hands that Feed Us; Lessons from a 10-mile diet (Viking/Penguin 2014) recounts her adventures in hyper-local eating and what she learned about food and farming as well as belonging and hope. For Show Notes and Transcript visit: https://www.thegreatsimplification.com/episode/21-vicki-robin

1 Juni 20221h 2min

Daniel Schmachtenberger: "Bend not Break #2: Maximum Power and Hyper Agents"

Daniel Schmachtenberger: "Bend not Break #2: Maximum Power and Hyper Agents"

On this episode we meet with founding member of The Consilience Project, Daniel Schmachtenberger. In the second of a four-part series, Nate and Daniel explore the relationship between energy, information, technology, the Superorganism, and the maximum power principle. How can we maximize returns on agency? Nate and Daniel explain the importance of hyper agents: those humans who have an outsized influence on what's happening in the world. About Daniel Schmachtenberger: Daniel Schmachtenberger is a founding member of The Consilience Project, aimed at improving public sensemaking and dialogue. The throughline of his interests has to do with ways of improving the health and development of individuals and society, with a virtuous relationship between the two as a goal. Towards these ends, he's had particular interest in the topics of catastrophic and existential risk, civilization and institutional decay and collapse as well as progress, collective action problems, social organization theories, and the relevant domains in philosophy and science. For Show Notes and Transcript visit: https://www.thegreatsimplification.com/episode/20-daniel-schmactenberger

25 Maj 20221h 51min

Populärt inom Vetenskap

p3-dystopia
svd-nyhetsartiklar
dumma-manniskor
allt-du-velat-veta
sexet
ufo-sverige
det-morka-psyket
kapitalet-en-podd-om-ekonomi
dumforklarat
rss-vetenskapsradion
rss-vetenskapspodden
rss-ufobortom-rimligt-tvivel
paranormalt-med-caroline-giertz
rss-vetenskapsradion-2
medicinvetarna
rss-arkeologi-historia
rss-i-hjarnan-pa-louise-epstein
bildningspodden
naturmorgon
barnpsykologerna