114. Sam Bowman - Are we *under-hyping* AI?

Google the phrase “AI over-hyped”, and you’ll find literally dozens of articles from the likes of Forbes, Wired, and Scientific American, all arguing that “AI isn’t really as impressive at it seems from the outside,” and “we still have a long way to go before we come up with *true* AI, don’t you know.”

Amusingly, despite the universality of the “AI is over-hyped” narrative, the statement that “We haven’t made as much progress in AI as you might think™️” is often framed as somehow being an edgy, contrarian thing to believe.

All that pressure not to over-hype AI research really gets to people — researchers included. And they adjust their behaviour accordingly: they over-hedge their claims, cite outdated and since-resolved failure modes of AI systems, and generally avoid drawing straight lines between points that clearly show AI progress exploding across the board. All, presumably, to avoid being perceived as AI over-hypers.

Why does this matter? Well for one, under-hyping AI allows us to stay asleep — to delay answering many of the fundamental societal questions that come up when widespread automation of labour is on the table. But perhaps more importantly, it reduces the perceived urgency of addressing critical problems in AI safety and AI alignment.

Yes, we need to be careful that we’re not over-hyping AI. “AI startups” that don’t use AI are a problem. Predictions that artificial general intelligence is almost certainly a year away are a problem. Confidently prophesying major breakthroughs over short timescales absolutely does harm the credibility of the field.

But at the same time, we can’t let ourselves be so cautious that we’re not accurately communicating the true extent of AI’s progress and potential. So what’s the right balance?

That’s where Sam Bowman comes in. Sam is a professor at NYU, where he does research on AI and language modeling. But most important for today’s purposes, he’s the author of a paper titled, “When combating AI hype, proceed with caution,” in which he explores a trend he calls under-claiming — a common practice among researchers that consists of under-stating the extent of current AI capabilities, and over-emphasizing failure modes in ways that can be (unintentionally) deceptive.

Sam joined me to talk about under-claiming and what it means for AI progress on this episode of the Towards Data Science podcast.

***

Intro music:

- Artist: Ron Gelinas

- Track Title: Daybreak Chill Blend (original mix)

- Link to Track: https://youtu.be/d8Y2sKIgFWc

***

Chapters:

• 2:15 Overview of the paper
• 8:50 Disappointing systems
• 13:05 Potential double standard
• 19:00 Moving away from multi-modality
• 23:50 Overall implications
• 28:15 Pressure to publish or perish
• 32:00 Announcement discrepancies
• 36:15 Policy angle
• 41:00 Recommendations
• 47:20 Wrap-up