Understanding the Most Viral Chart in Artificial Intelligence

We live in an era of charts that are going up and to the right. This image obviously describes the stock market, particularly any company whose business is adjacent to artificial intelligence. But beyond stocks, another sort of chart we keep seeing is of AI capabilities also going up and to the right. The most famous and viral of these comes from an organization called METR, which stands for Model Evaluation and Threat Research. The organization is focused on understanding the degree to which AI models can engage in autonomous, complex tasks. METR see this is as a particularly important benchmark, given the risk that AI could one day be engaged in recursive self improvement, taking humans out of the loop. But how do you really gauge a model's ability to do complex problems. And what is being measured for exactly? On this episode, we speak with METR's President Chris Painter as well as Joel Becker, a member of the technical staff who works on evaluation methods for the organization. We discuss both the mechanics and the philosophy of METR's work, and what it means when we see a a chart showing that Clause Opus 4.6 can do a task that would take a human nearly 12 hours.

Only http://Bloomberg.com subscribers can get the Odd Lots newsletter in their inbox each week, plus unlimited access to the site and app. Subscribe at bloomberg.com/subscriptions/oddlots

Subscribe to the Odd Lots Newsletter
Join the conversation: discord.gg/oddlots

See omnystudio.com/listener for privacy information.

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(1241)

NY Governor Kathy Hochul on Her One Year Data Center Moratorium

On July 14, New York Governor Kathy Hochul announced a one-year moratorium on new large data center projects in the state. It's the nation's first such pause at a time when anti-data center politics i...

15 Heinä 42min

Why AI Might Actually Create More Work for Lawyers

It seems obvious that among the many industries that AI might disrupt, the legal profession might face some of the most adverse outcomes. When clerical, research-based tasks like searching through dat...

13 Heinä 55min

The Korean Levered ETFs Shaking Markets All Around the World

Retail participation in the stock market is booming. And of course the biggest story in markets is the AI trade, which has created an incredible amount of demand for chips and memory. These two broad ...

10 Heinä 43min

One of the World's Largest Hedge Funds on Its 86x Growth in Token Spending

We've gone through a number of a technological revolutions in investing, whether it was the dawn of the high frequency trading era or the introduction of robotraders. When it comes to AI, the big ques...

9 Heinä 52min

These Are the Sharps Actually Making Money on Prediction Markets

Here's a couple things about prediction markets. A lot of it is pure gambling and speculation, much of it on things with very little economic relevance. Another fact is that in all likelihood, if you ...

6 Heinä 48min

How a Major Grocery Store Chain Can Dramatically Lower the Cost of Food

In June, grocery giant Aldi opened a store just off of Times Square in Manhattan. It's the company's first location in Midtown and, according to their US Chief Commercial Officer Scott Patton, Aldi ha...

3 Heinä 50min

What Dan Wang Saw on His Last Trip to China

There's this weird contradiction that hovers almost all conversations regarding the Chinese economy. On the one hand, the growth and rising material prosperity is undeniable. And of course, Chinese in...

2 Heinä 48min

Baidu's CFO on How It Became a Full-Stack AI Player

In the China tech space, Baidu is now a full-stack player in the AI industry. The company makes its own chips, has its own AI models (Ernie), its own cloud system, and it's integrating AI into its sel...

29 Kesä 47min