“How Well Does RL Scale?” by Toby_Ord

“How Well Does RL Scale?” by Toby_Ord

This is the latest in a series of essays on AI Scaling.
You can find the others on my site.

Summary: RL-training for LLMs scales surprisingly poorly. Most of its gains are from allowing LLMs to productively use longer chains of thought, allowing them to think longer about a problem. There is some improvement for a fixed length of answer, but not enough to drive AI progress. Given the scaling up of pre-training compute also stalled, we'll see less AI progress via compute scaling than you might have thought, and more of it will come from inference scaling (which has different effects on the world). That lengthens timelines and affects strategies for AI governance and safety.

The current era of improving AI capabilities using reinforcement learning (from verifiable rewards) involves two key types of scaling:

  1. Scaling the amount of compute used for RL during training
  2. Scaling [...]

---

Outline:

(09:12) How do these compare to pre-training scaling?

(13:42) Conclusion

---

First published:
October 22nd, 2025

Source:
https://forum.effectivealtruism.org/posts/TysuCdgwDnQjH3LyY/how-well-does-rl-scale

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(250)

“Let’s taboo the V-word” by lincolnq

“Let’s taboo the V-word” by lincolnq

“How long have you been v*g*n?” This is one of the most common icebreakers at animal protection events. It's a baseline assumption, and it mostly holds true: if you’re out advocating for animals not t...

14 Heinä 12min

“Giving What We Can’s first  YouTube Video is out now!” by JustinPortela

“Giving What We Can’s first YouTube Video is out now!” by JustinPortela

Hello! I'm Justin Portela. I got hired by GWWC to make YouTube videos after AI in Context did such a kickass job. My channel is using that same cinematic, high-production value beauty to talk about e...

9 Heinä 1min

“I’m never satisfied” by Ajeya

“I’m never satisfied” by Ajeya

Note: This post was crossposted from Planned Obsolescence by the Forum team, with the author's permission. The author may not see or respond to comments on this post. But we get the job done I was twe...

8 Heinä 6min

“Maybe do the thing you wish CEA would do” by alejoacelas 🔸

“Maybe do the thing you wish CEA would do” by alejoacelas 🔸

I used AI to fix transcription errors, rerrarange the ideas, and suggest tweaks to the title and some sentences. Three of the most exciting projects to come out of EA in recent years are, in a vague s...

8 Heinä 4min

“Mabye do the thing you wish CEA would do” by AlejoAcelas🔸

“Mabye do the thing you wish CEA would do” by AlejoAcelas🔸

I used AI to fix transcription errors, rerrarange the ideas, and suggest tweaks to the title and some sentences. Three of the most exciting projects to come out of EA in recent years are, in a vague s...

8 Heinä 4min

“Possible mistake EAs are making and shout out to Pause AI UK” by Michelle_Hutchinson

“Possible mistake EAs are making and shout out to Pause AI UK” by Michelle_Hutchinson

I think right now EAs might be making a significant mistake by paying insufficient attention to the political realm. As EAs we tend to figure out what's most impactful for us to work on and focus hard...

29 Kesä 6min

“Coming Around To Political Donations” by Jeff Kaufman 🔸

“Coming Around To Political Donations” by Jeff Kaufman 🔸

Five years ago I read a post on the EA Forum arguing that "election campaign contributions might be a way in which you can have a substantial impact as a small donor". It struck me as weird but plausi...

12 Kesä 4min

“animal welfare has an evidence problem” by matthes

“animal welfare has an evidence problem” by matthes

Why I stopped donating to animal welfare charities but feel more motivated than ever to redirect money and talent to the cause. I have wanted to write this post for a while. It is an uncomfortable thi...

6 Kesä 26min

Suosittua kategoriassa Yhteiskunta

olipa-kerran-otsikko
seitseman
siita-on-vaikea-puhua
i-dont-like-mondays
antin-palautepalvelu
poks
sita
hupiklubi
kaksi-aitia
mamma-mia
gogin-ja-janin-maailmanhistoria
kolme-kaannekohtaa
yopuolen-tarinoita-2
ihme-ja-kumma
loukussa
rss-palmujen-varjoissa
meidan-pitais-puhua
aikalisa
mystista
lahko