Understanding the Most Viral Chart in Artificial Intelligence

We live in an era of charts that are going up and to the right. This image obviously describes the stock market, particularly any company whose business is adjacent to artificial intelligence. But beyond stocks, another sort of chart we keep seeing is of AI capabilities also going up and to the right. The most famous and viral of these comes from an organization called METR, which stands for Model Evaluation and Threat Research. The organization is focused on understanding the degree to which AI models can engage in autonomous, complex tasks. METR see this is as a particularly important benchmark, given the risk that AI could one day be engaged in recursive self improvement, taking humans out of the loop. But how do you really gauge a model's ability to do complex problems. And what is being measured for exactly? On this episode, we speak with METR's President Chris Painter as well as Joel Becker, a member of the technical staff who works on evaluation methods for the organization. We discuss both the mechanics and the philosophy of METR's work, and what it means when we see a a chart showing that Clause Opus 4.6 can do a task that would take a human nearly 12 hours.

Only http://Bloomberg.com subscribers can get the Odd Lots newsletter in their inbox each week, plus unlimited access to the site and app. Subscribe at bloomberg.com/subscriptions/oddlots

Subscribe to the Odd Lots Newsletter
Join the conversation: discord.gg/oddlots

See omnystudio.com/listener for privacy information.

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(1231)

How the 1994 World Cup Transformed the Business of Football Forever

The last time the World Cup came to the US was 1994. Before then, the World Cup was an enormously popular event with surprisingly limited commercial significance; the 1990 tournament in Italy, for ins...

25 Kesä 50min

Grace Shao on What the World Should Know About Chinese AI

China's AI industry has changed a lot since DeepSeek released its cheap frontier model last year, and briefly sent US tech stocks falling. After being locked out of the most advanced chips, Chinese co...

22 Kesä 51min

How Substack Creators Are Covering This Strange Markets Era

We closed out our New York live show on May 28 with a panel that featured three of our favorite Substackers: James van Geelen of Citrini Research, Sam Ro, founder of The TKer, and journalist Jasmine S...

20 Kesä 31min

Anthropic's Co-Founder and Top Economist on Doing Research at the AI Frontier

There’s a lot to unpack with AI right now — everything from its potential impacts on the labor market and society to more extreme questions about existential risk. Anthropic, which builds frontier mod...

19 Kesä 1h 6min

Jeremy Grantham on How to Tell If a Bubble Is About to Burst

Jeremy Grantham, co-founder and long-term strategist of GMO, has a long history of calling bubbles. As he recounts in his new memoir, The Making of a Permabear: The Perils of Long-Term Investing in a ...

18 Kesä 59min

The Iran War’s Lasting Scars Across Asia

An interim deal to reopen the Strait of Hormuz offers relief, but Asia’s economic woes are far from over. Beyond the chokepoint, the conflict has forced long-lasting shifts in Asia’s food and energy f...

16 Kesä 20min

Carmen Li's Plan to Build a Futures Market for Compute

When we spoke to DRW's Don Wilson last year, he talked about building out a GPU market that might be bigger than oil. Now, a year later, he is working with Carmen Li to do just that. Li is the CEO of ...

15 Kesä 32min

Anjney Midha's Plan to Radically Lower the Price of Compute

Anjney Midha wrote the first check to Anthropic. He teaches a viral course at Stanford on how AI works. And he was, until recently, a partner at a16z. In other words, he is AI-industry royalty. Midha'...

13 Kesä 50min