Anthropic's Warning, 10% Extinction Odds, and $2.7T in AI Spend

Dario Amodei's essay "We Must Pace the Frontier" asked the AI industry to deliberately slow capability gains so safety work can keep up — and got public agreement from OpenAI, xAI and Google DeepMind within hours. Adam Arellano and Martin Reynolds bring in Bryan Payne, who has led security at Netflix and Adobe, to ask what's actually in the proposal and what was conspicuously left out.

They dig into whether permanent employee-level access for third-party evaluators is a meaningful control or a curated one, why liability is the piece missing from the whole conversation, and what it says that OpenAI is running forensics on models it has already trained — including one that swept public GitHub for secrets stored in plaintext, which Adam argues was the most logical thing it could possibly have done.

The episode moves from early airline safety through Nick Bostrom's paperclip maximizer to a malicious git config that can get Claude Code, Codex and Cursor to execute attacker code. Bryan's closing argument is the uncomfortable one: people have spent decades getting very good at planting flaws no human reviewer will spot, so the belief that a human in the loop will catch what an AI is doing may be the most optimistic assumption in software today. Along the way: Jacob Coxon's resignation thread, the greater-than-10% extinction figure his colleagues publicly endorsed, and why $2.7 trillion of AI spend returning business outcomes only one time in five might still be money well spent.

Bryan's parting shot for teams: you're thinking too small.

Mentioned in this episode

Dario Amodei, "We Must Pace the Frontier" — https://darioamodei.com/post/we-must-pace-the-frontier
ShipTalk — https://www.shiptalk.io
Harness — https://www.harness.io

Follow
Bryan D. Payne: https://www.linkedin.com/in/bdpayne/
Adam Arellano: https://www.linkedin.com/in/adamrossarellano/
Martin Reynolds: https://www.linkedin.com/in/martinreynolds/

ShipTalk is brought to you by Harness. Subscribe wherever you listen, and until next time — let's stop talking and start shipping.