“How Well Does RL Scale?” by Toby_Ord

“How Well Does RL Scale?” by Toby_Ord

This is the latest in a series of essays on AI Scaling.
You can find the others on my site.

Summary: RL-training for LLMs scales surprisingly poorly. Most of its gains are from allowing LLMs to productively use longer chains of thought, allowing them to think longer about a problem. There is some improvement for a fixed length of answer, but not enough to drive AI progress. Given the scaling up of pre-training compute also stalled, we'll see less AI progress via compute scaling than you might have thought, and more of it will come from inference scaling (which has different effects on the world). That lengthens timelines and affects strategies for AI governance and safety.

The current era of improving AI capabilities using reinforcement learning (from verifiable rewards) involves two key types of scaling:

  1. Scaling the amount of compute used for RL during training
  2. Scaling [...]

---

Outline:

(09:12) How do these compare to pre-training scaling?

(13:42) Conclusion

---

First published:
October 22nd, 2025

Source:
https://forum.effectivealtruism.org/posts/TysuCdgwDnQjH3LyY/how-well-does-rl-scale

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(250)

“Moral Imagination & Effective Altruism” by Toby_Ord

“Moral Imagination & Effective Altruism” by Toby_Ord

Michael Nielsen has a beautiful new essay on moral imagination: the ability humans have to 'develop and transmit new notions of good action, indeed, even new kinds of good'. As examples, he gives: Ha...

28 Elo 11min

“What if the third wave is a puddle?” by Morgan Fairless

“What if the third wave is a puddle?” by Morgan Fairless

The world of private philanthropy may be in for a period of large growth. News of Coefficient Giving increasing their funding for GiveWell to one billion in the near-term, and the general growth of th...

18 Elo 9min

“What would make us scale or stop? NOVAH’s pre-commitment before seeing the RCT results” by I.J.J., AlexisAt

“What would make us scale or stop? NOVAH’s pre-commitment before seeing the RCT results” by I.J.J., AlexisAt

TL;DR NOVAH (No Violence At Home) was incubated by Charity Entrepreneurship (now Ambitious Impact) in 2024 to test a promising idea: preventing intimate partner violence through edutainment, in our ca...

17 Elo 15min

“General capability - and capabilities generally - have no good y-axis” by Gregory Lewis🔸

“General capability - and capabilities generally - have no good y-axis” by Gregory Lewis🔸

BLUF: To determine whether AI is ‘improving exponentially’, ‘hitting the wall’, or any other claim which involves a quantity or magnitude (e.g. ‘This model was a big leap/small increment’). We need a...

8 Elo 1h 7min

“Never born, then maybe died twice” by NickLaing

“Never born, then maybe died twice” by NickLaing

“Your balls are baked aren’t they, mate”. He would not have been bornThanks Lyndon, that describes the situation. Like almost 1 in 10 modern men, my sperm weren’t up to the task - at least not the old...

1 Elo 10min

“Announcing: Cluelessness Critiques Competition” by Toby Tremlett🔹, Will Aldred

“Announcing: Cluelessness Critiques Competition” by Toby Tremlett🔹, Will Aldred

TL;DR We’re hosting an essay competition to elicit responses to Anthony DiGiovanni's sequence, ‘The challenge of unawareness for impartial altruist action guidance’—submit your entries by August 14th...

30 Heinä 13min

“Spiro: an update 2.5 years on and a fundraising ask for expansion” by Habiba Banu

“Spiro: an update 2.5 years on and a fundraising ask for expansion” by Habiba Banu

Summary Back in November 2023 I posted here to launch Spiro and raise our first 198 thousand dollars. Two and a half years later this is an update and a fundraiser for the next step. The short version...

26 Heinä 17min

“Let’s taboo the V-word” by lincolnq

“Let’s taboo the V-word” by lincolnq

“How long have you been v*g*n?” This is one of the most common icebreakers at animal protection events. It's a baseline assumption, and it mostly holds true: if you’re out advocating for animals not t...

14 Heinä 12min

Suosittua kategoriassa Yhteiskunta

i-dont-like-mondays
olipa-kerran-otsikko
hupiklubi
siita-on-vaikea-puhua
download
uutiscast
sita
poks
seitseman
antin-palautepalvelu
rss-murhan-anatomia
gogin-ja-janin-maailmanhistoria
kaksi-aitia
yopuolen-tarinoita-2
kolme-kaannekohtaa
mamma-mia
ihme-ja-kumma
aikalisa
loukussa
rss-haudattu