Understanding the Most Viral Chart in Artificial Intelligence

We live in an era of charts that are going up and to the right. This image obviously describes the stock market, particularly any company whose business is adjacent to artificial intelligence. But beyond stocks, another sort of chart we keep seeing is of AI capabilities also going up and to the right. The most famous and viral of these comes from an organization called METR, which stands for Model Evaluation and Threat Research. The organization is focused on understanding the degree to which AI models can engage in autonomous, complex tasks. METR see this is as a particularly important benchmark, given the risk that AI could one day be engaged in recursive self improvement, taking humans out of the loop. But how do you really gauge a model's ability to do complex problems. And what is being measured for exactly? On this episode, we speak with METR's President Chris Painter as well as Joel Becker, a member of the technical staff who works on evaluation methods for the organization. We discuss both the mechanics and the philosophy of METR's work, and what it means when we see a a chart showing that Clause Opus 4.6 can do a task that would take a human nearly 12 hours.

Only http://Bloomberg.com subscribers can get the Odd Lots newsletter in their inbox each week, plus unlimited access to the site and app. Subscribe at bloomberg.com/subscriptions/oddlots

Subscribe to the Odd Lots Newsletter
Join the conversation: discord.gg/oddlots

See omnystudio.com/listener for privacy information.

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(1307)

How Wall Street Is Bracing For the Post-Jensen Huang Nvidia Era

Nvidia's post-Jensen Huang era has just begun and there are so many open questions about the company's future. Who will the board choose to be Nvidia's next CEO? Will the government crack down on the ...

14 Nov 35min

Why a $2 Billion Law Firm Suddenly Collapsed

At the end of October, seemingly overnight, a major law firm called Weil Gotshal announced it was winding down after a number of partners departed and talks of a merger with another firm failed. So ho...

13 Nov 58min

What Nvidia's Megaspeed Scandal Is Really All About

Nvidia CEO Jensen Huang’s extramarital relationship with a woman named Alice Huang, whom US intelligence reportedly identified as a Chinese intelligence asset, is one of the biggest stories in the wor...

12 Nov 54min

The Perfect Storm That Clobbered Hedge Funds

Even before the US midterms, the AI trade was weakening. Just look at all the headlines from October in the hedge fund world — it was one of the bloodiest periods the industry has faced in recent memo...

11 Nov 51min

This Is How New York City Takes Out Its Trash

According to Joshua Goodman, trash collection in New York is one of the last services the city offers that is not yet touched by neoliberalism. What he means is that it's the last free and unlimited c...

9 Nov 52min

The Macro Implications of the Blue Tsunami

We don't need to tell you that the midterm elections were historic: The Democrats won bigger than even they imagined. But as the dust settles, an open question we are puzzling over is the market's ong...

6 Nov 37min

James Bosworth on What Comes After Lula's Stunning Win

When we last talked to James Bosworth, it seemed like an “orange wave” was going to take over Latin America and further realign the region’s interests with those of the Trump administration. But with ...

5 Nov 32min

An Election Day Special With Nate Silver

Nate Silver tells us he is calling this midterms the “rock and a hard place election” — the background conditions are great for Democrats to win both the House and the Senate, but for that to come to ...

3 Nov 42min