80,000 Hours Podcast30 Loka 2025

#226 – Holden Karnofsky on unexploited opportunities to make AI safer — and all his AGI takes

For years, working on AI safety usually meant theorising about the ‘alignment problem’ or trying to convince other people to give a damn. If you could find any way to help, the work was frustrating and low feedback.

According to Anthropic’s Holden Karnofsky, this situation has now reversed completely.

There are now large amounts of useful, concrete, shovel-ready projects with clear goals and deliverables. Holden thinks people haven’t appreciated the scale of the shift, and wants everyone to see the large range of ‘well-scoped object-level work’ they could personally help with, in both technical and non-technical areas.

Video, full transcript, and links to learn more: https://80k.info/hk25

In today’s interview, Holden — previously cofounder and CEO of Open Philanthropy (now Coefficient Giving) — lists 39 projects he’s excited to see happening, including:

Training deceptive AI models to study deception and how to detect it
Developing classifiers to block jailbreaking
Implementing security measures to stop ‘backdoors’ or ‘secret loyalties’ from being added to models in training
Developing policies on model welfare, AI-human relationships, and what instructions to give models
Training AIs to work as alignment researchers

And that’s all just stuff he’s happened to observe directly, which is probably only a small fraction of the options available.

Holden makes a case that, for many people, working at an AI company like Anthropic will be the best way to steer AGI in a positive direction. He notes there are “ways that you can reduce AI risk that you can only do if you’re a competitive frontier AI company.” At the same time, he believes external groups have their own advantages and can be equally impactful.

Critics worry that Anthropic’s efforts to stay at that frontier encourage competitive racing towards AGI — significantly or entirely offsetting any useful research they do. Holden thinks this seriously misunderstands the strategic situation we’re in — and explains his case in detail with host Rob Wiblin.

Chapters:

Cold open (00:00:00)
Holden is back! (00:02:26)
An AI Chernobyl we never notice (00:02:56)
Is rogue AI takeover easy or hard? (00:07:32)
The AGI race isn't a coordination failure (00:17:48)
What Holden now does at Anthropic (00:28:04)
The case for working at Anthropic (00:30:08)
Is Anthropic doing enough? (00:40:45)
Can we trust Anthropic, or any AI company? (00:43:40)
How can Anthropic compete while paying the “safety tax”? (00:49:14)
What, if anything, could prompt Anthropic to halt development of AGI? (00:56:11)
Holden's retrospective on responsible scaling policies (00:59:01)
Overrated work (01:14:27)
Concrete shovel-ready projects Holden is excited about (01:16:37)
Great things to do in technical AI safety (01:20:48)
Great things to do on AI welfare and AI relationships (01:28:18)
Great things to do in biosecurity and pandemic preparedness (01:35:11)
How to choose where to work (01:35:57)
Overrated AI risk: Cyberattacks (01:41:56)
Overrated AI risk: Persuasion (01:51:37)
Why AI R&D is the main thing to worry about (01:55:36)
The case that AI-enabled R&D wouldn't speed things up much (02:07:15)
AI-enabled human power grabs (02:11:10)
Main benefits of getting AGI right (02:23:07)
The world is handling AGI about as badly as possible (02:29:07)
Learning from targeting companies for public criticism in farm animal welfare (02:31:39)
Will Anthropic actually make any difference? (02:40:51)
“Misaligned” vs “misaligned and power-seeking” (02:55:12)
Success without dignity: how we could win despite being stupid (03:00:58)
Holden sees less dignity but has more hope (03:08:30)
Should we expect misaligned power-seeking by default? (03:15:58)
Will reinforcement learning make everything worse? (03:23:45)
Should we push for marginal improvements or big paradigm shifts? (03:28:58)
Should safety-focused people cluster or spread out? (03:31:35)
Is Anthropic vocal enough about strong regulation? (03:35:56)
Is Holden biased because of his financial stake in Anthropic? (03:39:26)
Have we learned clever governance structures don't work? (03:43:51)
Is Holden scared of AI bioweapons? (03:46:12)
Holden thinks AI companions are bad news (03:49:47)
Are AI companies too hawkish on China? (03:56:39)
The frontier of infosec: confidentiality vs integrity (04:00:51)
How often does AI work backfire? (04:03:38)
Is AI clearly more impactful to work in? (04:18:26)
What's the role of earning to give? (04:24:54)

This episode was recorded on July 25 and 28, 2025.

Video editing: Simon Monsour, Luke Monsour, Dominic Armstrong, and Milo McGuire
Audio engineering: Milo McGuire, Simon Monsour, and Dominic Armstrong
Music: CORBIT
Coordination, transcriptions, and web: Katy Moore

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(334)

Rob & Howie on what we do and don't know about 2019-nCoV

Two 80,000 Hours researchers, Robert Wiblin and Howie Lempel, record an experimental bonus episode about the new 2019-nCoV virus.See this list of resources, including many discussed in the episode, to...

3 Helmi 20201h 18min

#68 - Will MacAskill on the paralysis argument, whether we're at the hinge of history, & his new priorities

You’re given a box with a set of dice in it. If you roll an even number, a person's life is saved. If you roll an odd number, someone else will die. Each time you shake the box you get $10. Should you...

24 Tammi 20203h 25min

#44 Classic episode - Paul Christiano on finding real solutions to the AI alignment problem

Rebroadcast: this episode was originally released in October 2018. Paul Christiano is one of the smartest people I know. After our first session produced such great material, we decided to do a seco...

15 Tammi 20203h 51min

#33 Classic episode - Anders Sandberg on cryonics, solar flares, and the annual odds of nuclear war

Rebroadcast: this episode was originally released in May 2018. Joseph Stalin had a life-extension program dedicated to making himself immortal. What if he had succeeded? According to Bryan Caplan ...

8 Tammi 20201h 25min

#17 Classic episode - Will MacAskill on moral uncertainty, utilitarianism & how to avoid being a moral monster

Rebroadcast: this episode was originally released in January 2018. Immanuel Kant is a profoundly influential figure in modern philosophy, and was one of the earliest proponents for universal democracy...

31 Joulu 20191h 52min

#67 – David Chalmers on the nature and ethics of consciousness

What is it like to be you right now? You're seeing this text on the screen, smelling the coffee next to you, and feeling the warmth of the cup. There’s a lot going on in your head — your conscious exp...

16 Joulu 20194h 41min

#66 – Peter Singer on being provocative, effective altruism, & how his moral views have changed

In 1989, the professor of moral philosophy Peter Singer was all over the news for his inflammatory opinions about abortion. But the controversy stemmed from Practical Ethics — a book he’d actually rel...

5 Joulu 20192h 1min

#65 – Ambassador Bonnie Jenkins on 8 years pursuing WMD arms control, & diversity in diplomacy

"…it started when the Soviet Union fell apart and there was a real desire to ensure security of nuclear materials and pathogens, and that scientists with [WMD-related] knowledge could get paid so that...

19 Marras 20191h 40min