“General capability - and capabilities generally - have no good y-axis” by Gregory Lewis🔸

“General capability - and capabilities generally - have no good y-axis” by Gregory Lewis🔸

BLUF:

  • To determine whether AI is ‘improving exponentially’, ‘hitting the wall’, or any other claim which involves a quantity or magnitude (e.g. ‘This model was a big leap/small increment’). We need a good y-axis: an interval scale of AI capability which means +1 unit always represents the same degree of ‘how much better’, in the same way +1 degree Celsius is always the same amount of ‘how much hotter’.
  • Yet there is no good y-axis for AI capability. All our measures are of something related-to but clearly not identical-with it, thus ‘true’ AI capability can be a funhouse-mirror reflection of whatever was measured. Specifically:
    • Benchmark score: One small step in benchmark score can be a giant leap in capability, or the opposite, or whatever else. (My 6/10 vs. your 4/10 ≠ I’m 50% better at maths than you).
    • Elo et al: Can give a real y-axis in terms of winning chances, but doesn’t translate outside of beating others. (Going from 50% to 73% to 88% chance to get a higher score than you on a maths test ≠ gaining 0 → 1 → 2 units of maths ability over you)
    • Epoch Capabilities Index: Analogous to IQ, so [...]

---

Outline:

(03:05) Introduction

(04:26) Both a poor reflection and a dark glass

(07:51) Human benchmarking also has a y-axis problem

(12:27) A metrological elegy

(12:49) The base case: benchmarks (cf. exams)

(13:30) Elo et al.

(16:25) (And maybe not quite 'game ability ≡ winning games', after all?)

(18:28) ECI (cf. IQ)

(20:56) Intervals Rarely True

(25:01) Measure endogeneity

(31:12) Forking IRT

(34:45) (Dimensions of being, and beating, a bat)

(39:40) Prediction (cf. chronometry)

(41:58) Time horizons

(44:34) Human "capability" is also exponential in time horizon

(47:38) Intuitive/interpretative prelude

(52:56) Time horizons and ECI share an axis kink

(57:13) Perhaps money, as a measure, stinks the least

(01:01:07) Finale: AI as normal epistemics

(01:06:48) Acknowledgements

The original text contained 33 footnotes which were omitted from this narration.

---

First published:
August 4th, 2026

Source:
https://forum.effectivealtruism.org/posts/CQvdadxjCpd7i7kjA/general-capability-and-capabilities-generally-have-no-good-y

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(250)

“Never born, then maybe died twice” by NickLaing

“Never born, then maybe died twice” by NickLaing

“Your balls are baked aren’t they, mate”. He would not have been bornThanks Lyndon, that describes the situation. Like almost 1 in 10 modern men, my sperm weren’t up to the task - at least not the old...

1 Elo 10min

“Announcing: Cluelessness Critiques Competition” by Toby Tremlett🔹, Will Aldred

“Announcing: Cluelessness Critiques Competition” by Toby Tremlett🔹, Will Aldred

TL;DR We’re hosting an essay competition to elicit responses to Anthony DiGiovanni's sequence, ‘The challenge of unawareness for impartial altruist action guidance’—submit your entries by August 14th...

30 Heinä 13min

“Spiro: an update 2.5 years on and a fundraising ask for expansion” by Habiba Banu

“Spiro: an update 2.5 years on and a fundraising ask for expansion” by Habiba Banu

Summary Back in November 2023 I posted here to launch Spiro and raise our first 198 thousand dollars. Two and a half years later this is an update and a fundraiser for the next step. The short version...

26 Heinä 17min

“Let’s taboo the V-word” by lincolnq

“Let’s taboo the V-word” by lincolnq

“How long have you been v*g*n?” This is one of the most common icebreakers at animal protection events. It's a baseline assumption, and it mostly holds true: if you’re out advocating for animals not t...

14 Heinä 12min

“Giving What We Can’s first  YouTube Video is out now!” by JustinPortela

“Giving What We Can’s first YouTube Video is out now!” by JustinPortela

Hello! I'm Justin Portela. I got hired by GWWC to make YouTube videos after AI in Context did such a kickass job. My channel is using that same cinematic, high-production value beauty to talk about e...

9 Heinä 1min

“I’m never satisfied” by Ajeya

“I’m never satisfied” by Ajeya

Note: This post was crossposted from Planned Obsolescence by the Forum team, with the author's permission. The author may not see or respond to comments on this post. But we get the job done I was twe...

8 Heinä 6min

“Maybe do the thing you wish CEA would do” by alejoacelas 🔸

“Maybe do the thing you wish CEA would do” by alejoacelas 🔸

I used AI to fix transcription errors, rerrarange the ideas, and suggest tweaks to the title and some sentences. Three of the most exciting projects to come out of EA in recent years are, in a vague s...

8 Heinä 4min

Suosittua kategoriassa Yhteiskunta

olipa-kerran-otsikko
seitseman
i-dont-like-mondays
siita-on-vaikea-puhua
sita
poks
antin-palautepalvelu
uutiscast
gogin-ja-janin-maailmanhistoria
kaksi-aitia
mamma-mia
kolme-kaannekohtaa
hupiklubi
yopuolen-tarinoita-2
rss-palmujen-varjoissa
aikalisa
ihme-ja-kumma
loukussa
rss-haudattu
meidan-pitais-puhua