Ten: Holden Karnofsky on dozens of opportunities to make AI safer lying on the table — and all his AGI takes

For years, AI safety work mostly meant theorising about the ‘alignment problem’ or trying to convince people to give a damn. If you could find a way to help, the work was frustrating and low-feedback.

That situation has now reversed completely.

According to Holden Karnofsky — cofounder and former CEO of Open Philanthropy (now Coefficient Giving), now working at Anthropic — there's an overwhelming amount of concrete, useful safety work that needs doing in both technical and nontechnical areas, and nowhere near enough people to do it.

In this conversation alone, Holden lists 39 projects he’s excited about, including:

• Training deceptive AI models in order to study deception and how to detect it
• Developing classifiers to block jailbreaking
• Implementing security measures to stop ‘backdoors’ and ‘secret loyalties’ from being added to models during training
• Developing policies on model welfare, AI-human relationships, and what instructions to give models
• Training AIs to work as alignment researchers

And that’s just what he’s observed directly, likely a small fraction of what’s available.

All this low-hanging fruit is part of why he joined Anthropic this year. (Though his wife is cofounder and president of the company, giving him a big financial stake in its success — and making it impossible for him to be seen as independent, no matter where he works.)

Holden argues that for many people, working at a frontier AI company is the highest-impact way to steer artificial general intelligence (AGI) — by developing cheap safety tools other companies might actually adopt, prototyping policies that regulators could mandate, and generating hard data about what advanced AI can really do. But he's clear that external groups have distinct advantages and can be equally valuable.

Critics worry that Anthropic’s efforts to stay at that frontier encourage competitive racing towards AGI, significantly or entirely offsetting any useful research they do.

But Holden thinks this seriously misunderstands the current strategic situation. He believes the problem isn't that everyone wants to slow down but can't coordinate. Many major players simply don't believe the risks are real (or don't care about them, even if they do), don't want to slow down, and would be thrilled if a competitor dropped out — because they’d have a better chance at ‘winning.’

Host Rob Wiblin and Holden discuss all of this and much more, including:

• Why an 'AI Chernobyl' might happen without us ever noticing
• Holden’s case for working at Anthropic — and whether the company is doing enough for safety
• What, if anything, could prompt Anthropic to halt AGI development
• Why AI R&D is the main thing to worry about — and why fears about cyberattacks and persuasion are overrated
• How the world is handling AGI about as badly as possible
• Relevant lessons learned from targeting companies for public criticism over farm animal welfare
• How humanity could succeed without dignity, and win despite being stupid
• Whether AI companies are too hawkish on China
• The frontier of information security: confidentiality vs integrity

Learn more and read the full transcript on the 80,000 Hours website.

This episode was originally released in October 2025.

Chapters:

• Cold open (00:00:00)
• Holden is back! (00:02:28)
• An AI Chernobyl we never notice (00:02:58)
• Is rogue AI takeover easy or hard? (00:07:39)
• The AGI race isn't a coordination failure (00:18:01)
• What Holden now does at Anthropic (00:28:30)
• The case for working at Anthropic (00:30:38)
• Is Anthropic doing enough? (00:41:30)
• Can we trust Anthropic, or any AI company? (00:44:30)
• How can Anthropic compete while paying the “safety tax”? (00:50:11)
• What, if anything, could prompt Anthropic to halt development of AGI? (00:57:13)
• Holden's retrospective on responsible scaling policies (01:00:04)
• Overrated work (01:15:45)
• Concrete shovel-ready projects Holden is excited about (01:17:58)
• Great things to do in technical AI safety (01:22:12)
• Great things to do on AI welfare and AI relationships (01:29:53)
• Great things to do in biosecurity and pandemic preparedness (01:36:51)
• How to choose where to work (01:37:37)
• Overrated AI risk: Cyberattacks (01:43:38)
• Overrated AI risk: Persuasion (01:53:28)
• Why AI R&D is the main thing to worry about (01:57:31)
• The case that AI-enabled R&D wouldn't speed things up much (02:09:30)
• AI-enabled human power grabs (02:13:26)
• Main benefits of getting AGI right (02:26:04)
• The world is handling AGI about as badly as possible (02:31:44)
• Learning from targeting companies for public criticism in farm animal welfare (02:34:18)
• Will Anthropic actually make any difference? (02:43:43)
• “Misaligned” vs “misaligned and power-seeking” (02:58:23)
• Success without dignity: how we could win despite being stupid (03:04:16)
• Holden sees less dignity but has more hope (03:12:00)
• Should we expect misaligned power-seeking by default? (03:19:43)
• Will reinforcement learning make everything worse? (03:27:36)
• Should we push for marginal improvements or big paradigm shifts? (03:32:54)
• Should safety-focused people cluster or spread out? (03:35:32)
• Is Anthropic vocal enough about strong regulation? (03:39:55)
• Is Holden biased because of his financial stake in Anthropic? (03:43:30)
• Have we learned clever governance structures don't work? (03:47:57)
• Is Holden scared of AI bioweapons? (03:50:20)
• Holden thinks AI companions are bad news (03:53:58)
• Are AI companies too hawkish on China? (04:00:53)
• The frontier of infosec: confidentiality vs integrity (04:05:06)
• How often does AI work backfire? (04:07:55)
• Is AI clearly more impactful to work in? (04:22:50)
• What's the role of earning to give? (04:29:22)

Video editing: Simon Monsour, Luke Monsour, Dominic Armstrong, and Milo McGuire
Audio engineering: Milo McGuire, Simon Monsour, and Dominic Armstrong
Music: CORBIT
Coordination, transcriptions, and web: Katy Moore