Understanding the Most Viral Chart in Artificial Intelligence

We live in an era of charts that are going up and to the right. This image obviously describes the stock market, particularly any company whose business is adjacent to artificial intelligence. But beyond stocks, another sort of chart we keep seeing is of AI capabilities also going up and to the right. The most famous and viral of these comes from an organization called METR, which stands for Model Evaluation and Threat Research. The organization is focused on understanding the degree to which AI models can engage in autonomous, complex tasks. METR see this is as a particularly important benchmark, given the risk that AI could one day be engaged in recursive self improvement, taking humans out of the loop. But how do you really gauge a model's ability to do complex problems. And what is being measured for exactly? On this episode, we speak with METR's President Chris Painter as well as Joel Becker, a member of the technical staff who works on evaluation methods for the organization. We discuss both the mechanics and the philosophy of METR's work, and what it means when we see a a chart showing that Clause Opus 4.6 can do a task that would take a human nearly 12 hours.

Only http://Bloomberg.com subscribers can get the Odd Lots newsletter in their inbox each week, plus unlimited access to the site and app. Subscribe at bloomberg.com/subscriptions/oddlots

Subscribe to the Odd Lots Newsletter
Join the conversation: discord.gg/oddlots

See omnystudio.com/listener for privacy information.

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(1274)

There's a Mind-Boggling Number of Rich People in America

In 2014, the economists Owen Zidar and Eric Zwick were asked by the Treasury to conduct a study on the tax burden of private business owners. It was tricky to figure out how much these sorts of busine...

18 Sep 59min

What Francis Fukuyama Is Seeing at 'The End of History'

Francis Fukuyama's essay “The End of History?” was published in the summer of 1989, a few months before the fall of the Berlin Wall. The essay's argument — that liberal democracy would not only defeat...

17 Sep 54min

OpenAI President Greg Brockman on Doing Business in the Wake of Hugging Face

According to OpenAI President Greg Brockman, the models that escaped their sandbox and hacked into Hugging Face's servers had yet to go through alignment training. That they were able to break free of...

14 Sep 1h 2min

Robert Friedland on the World's Monumental Shortage of Copper

Copper demand is booming, with the metal surging another 15% this year. Copper demand has always been tightly linked to economic activity, but now it's in overdrive, thanks to data centers, electric c...

12 Sep 1h 9min

Why Bridgewater's CIO Says AI's Human Extinction Risk Is Real

Greg Jensen was one of the earliest backers of both OpenAI and Anthropic, and at Bridgewater Associates, where he is the managing chief investment officer, he leads the hedge fund’s AI strategy. As an...

11 Sep 1h 5min

The Rise of Organized Retail Crime at Big Box Stores

If you walk into a local pharmacy in New York City, more likely than not, lots of the merchandise is locked behind closed doors as a measure to mitigate shoplifting. So how big is the “retail shrink” ...

10 Sep 52min

Why Money Launderers Love $100 Bills

Hardly anyone nowadays seems to carry much cash, never mind carrying around a bunch of $100 bills. So why does the amount of physical cash in circulation — especially big denominations like the $100 b...

7 Sep 54min

Why Laser Beams Are the Hottest New Tech in Defense

Our guest today is Colonel Sanders. More precisely, we speak with Wayne Sanders, a retired US army colonel who is currently senior defense research analyst for Bloomberg Intelligence. During his milit...

4 Sep 53min