#325 Phelim Brady: Why AI's Future Depends on Human Judgement
Eye On A.I.9 Maalis

#325 Phelim Brady: Why AI's Future Depends on Human Judgement

AI often looks fully automated. But behind the scenes, a huge amount of human judgment is shaping how these systems actually work.

In this episode, Craig Smith speaks with Phelim Bradley, co-founder and CEO of Prolific, a platform that connects millions of real people with researchers and AI labs to evaluate and improve AI systems.

They explore the hidden human layer behind modern AI, why traditional benchmarks are becoming less reliable, and why AI companies increasingly rely on real human feedback to measure model performance in the real world.

Phelim also explains how demographic differences influence how models are evaluated, why human judgment remains critical even as AI improves, and how the collaboration between humans and AI will shape the next phase of development.

This conversation reveals the human backbone behind today's AI systems.


Stay Updated:

Craig Smith on X: https://x.com/craigss

Eye on A.I. on X: https://x.com/EyeOn_AI

(00:00) Preview and Intro

(02:45) Founding Prolific And Early Pain Points

(06:30) From Mechanical Turk To Representativeness

(09:55) Academic Research And AI Use Cases Split

(13:40) Vetting Real Participants And Fighting Fraud

(17:45) Scale, Community Growth, And Talent Mix

(22:00) High-Complexity Projects Over Commoditised Labeling

(26:40) Measuring Model Persuasion With Live Conversations

(30:20) Demographic-Aware Model Preference Benchmarks

(34:10) The Rise Of Human Evaluation Over Benchmarks

(38:00) Enterprise Model Choice And Continuous Evaluation

(42:00) Why Humans Won't Disappear From The Loop





Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(352)

More Customers Chose the AI Agent Than Anyone Expected | Tom Chen, Aircall

More Customers Chose the AI Agent Than Anyone Expected | Tom Chen, Aircall

Every time you hit a phone tree or a chatbot with canned answers, you're experiencing the gap between what AI can already do and what most companies are still delivering. Craig Smith sits down with To...

4 Kesä 56min

Why the Future of AI Isn't Just Bigger Models. It's Models That Evolve | Risto Miikkulainen of Cognizant

Why the Future of AI Isn't Just Bigger Models. It's Models That Evolve | Risto Miikkulainen of Cognizant

Most AI systems follow a gradient, a mathematical slope that tells them exactly how to improve, step by step, toward a known goal. Neuroevolution doesn't follow any gradient. Instead, it runs hundreds...

2 Kesä 1h 4min

How AI Is Reinventing Elder Care | Chia-Lin Simmons of LogicMark

How AI Is Reinventing Elder Care | Chia-Lin Simmons of LogicMark

One in four people over 65 will experience a fall, and for most of them, the technology designed to help is a device that hasn't meaningfully changed since the 1980s. Chia-Lin Simmons, CEO of LogicMar...

1 Kesä 53min

The App of the Future Is Voice — Not a Screen. Mitel's CTO Luiz Domingos Explains Why.

The App of the Future Is Voice — Not a Screen. Mitel's CTO Luiz Domingos Explains Why.

Luiz Domingos has spent 25 years watching enterprise communications evolve, from IP telephony to cloud to AI, and his assessment of where things stand now is unusually concrete. Companies have moved p...

28 Touko 54min

Is ChatGPT Conscious? A Pioneer of AI Explains | Dr. Terry Sejnowski

Is ChatGPT Conscious? A Pioneer of AI Explains | Dr. Terry Sejnowski

A fly with 100,000 neurons can fly, find food, and reproduce. A $100 million supercomputer cannot. Dr. Terry Sejnowski used that observation to silence a room full of MIT AI researchers in the 1980s, ...

28 Touko 56min

Your Child's Data Profile Starts Before They're Born | Eamonn Maguire of Proton

Your Child's Data Profile Starts Before They're Born | Eamonn Maguire of Proton

Your child's data profile doesn't start when they get their first phone. It starts before they're born, the moment a parent emails a gynecologist or visits a fertility clinic website. That's the core ...

28 Touko 55min

Training AI Models Without a Billion-Dollar Data Center | Steffen Cruz of Macrocosmos

Training AI Models Without a Billion-Dollar Data Center | Steffen Cruz of Macrocosmos

Training a frontier AI model today requires hundreds of thousands of GPUs, months of compute time, and a budget that only a handful of companies on earth can afford. Steffen Cruz, co-founder and CTO o...

25 Touko 47min

The Single Biggest Barrier to AI Adoption Isn't the Technology — It's This | Errol Gardner of EY

The Single Biggest Barrier to AI Adoption Isn't the Technology — It's This | Errol Gardner of EY

Errol Gardner has spent 35 years advising the world's largest organizations through major technology transitions, and his assessment of where enterprise agentic AI actually stands is one of the most g...

22 Touko 54min