Raw Data with Rob Collie16 Des 2025

The Apparent Meaninglessness of AI Benchmarks, plus How to Explain AI Opportunities to Others

Every week brings a new AI benchmark. Higher scores. Bigger claims. Louder voices insisting this changes everything. And yet, when you put AI in front of a real business problem, none of that noise seems to help. In this episode, Rob and Justin dig into why AI benchmarks often feel strangely meaningless in practice and why that disconnect is the point. Benchmarks aren't useless. They're just answering a different question than the one most businesses are asking.

This isn't just random conjecture either. Rob walks through what he's learned building actual AI workflows and why a twenty percent improvement on a leaderboard rarely translates into anything you can feel on the job. They talk about why model choice usually isn't the bottleneck, why swapping models should be easy if you've built things the right way, and why the most successful AI work rarely shows up as a flashy demo. Most of the value is happening quietly, off-screen, inside systems that look a lot more like normal software than artificial intelligence.

Rob and Justin also talk about why explaining AI is often harder than building it. The first demo people see tends to stick, even when it's the wrong one. Consumer AI feels magical. Business AI face plants unless it's built with intent, structure, and real context. This episode gives leaders better language for that gap, without hype or panic. If you're done chasing benchmarks and just want a way to think about AI that survives contact with reality, this episode's for you.

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(245)

AI and the Art of Building Backwards

There are two ways to build backwards. One starts with the answer you actually want and works its way toward the plumbing. The other starts with the plumbing and hopes the answer is in there somewhere...

29 Sep 29min

AI and the Rule Nobody Wrote

Rob has a lot of Claude folders now. One knows the book. Another knows the company planning. Others have picked up meeting notes, briefing docs, LinkedIn ideas, strategy work, and whatever else was us...

22 Sep 35min

Your Favorite AI Has a Moat Problem

For a minute there, it looked like the AI wars would come down to who built the smartest model. Rob's not buying that anymore. Take Claude. Rob's increasingly convinced that what makes Anthropic so st...

15 Sep 25min

Knowledge Is Having Its Data Moment

For years, businesses learned that having data and being able to use it are two very different things. We cleaned it, structured it, modeled it, argued about which version was the truth, and built ent...

1 Sep 22min

What is "Botsitting?" (plus: Data Agents Continue to Blow Our Minds)

There's a new word for what your team is doing with the AI you bought them. You're not going to love it. "Botsitting." Babysitting, but for bots. Journalists are suddenly all over it, and not one of t...

18 Aug 38min

Power BI Veterans React to the Fair Game AI Book

For most of tech history, the new thing belonged to the new people. Then AI showed up and ruined a perfectly good pattern. Because the smartest model on the planet can know darn near everything and st...

12 Aug 1h 24min

The Crafter's Edge with Pedro Rossello

There have always been people who can't help themselves. Give them a clunky process, a messy spreadsheet, or a problem nobody owns, and they're already halfway to building something better. Rob calls ...

4 Aug 1h 24min

Everyone's Priority. Nobody's Job

AI headlines have already moved on to the deep end. Most businesses haven't. Every week brings another headline about what's next for AI. Build your own model. Train your own LLM. Customize everything...

14 Jul 32min