#298 Ryan Kolln: How Appen Trains the World's Most Powerful AI Models
Eye On A.I.6 Marras 2025

#298 Ryan Kolln: How Appen Trains the World's Most Powerful AI Models

This episode is sponsored by AGNTCY. Unlock agents at scale with an open Internet of Agents.

Visit https://agntcy.org/ and add your support.


How do the world's most powerful AI models get trained and trusted at scale, and what does that really take from data to deployment?

In this episode, Appen CEO Ryan Kolln joins Eye on AI to unpack how rigorous human evaluation, culturally aware data, and model-based judges come together to raise real-world performance.

In this episode of Eye on AI, host Craig Smith speaks with Ryan Kolln, CEO of Appen, about building evaluation systems that go beyond static benchmarks to measure usefulness, safety, and reliability in production. They explore how human raters and AI evaluators work in tandem, why localization matters across regions and domains, and how quality controls keep feedback signals trustworthy for training and post-training.

Ryan explains how evaluation feeds reinforcement strategies, where rubric-driven human judgments inform reward models, and how enterprises can stand up secure workflows for sensitive use cases. He also discusses emerging needs around sovereign models, domain-specific testing, and the shift from general chat to agentic workflows that operate inside real business systems.

Learn how leading teams design human-in-the-loop evaluation, when to route judgments from models back to expert reviewers, how to capture cultural nuance without losing universal guardrails, and how to build an evaluation stack that scales from early prototypes to production AI.


Stay Updated:
Craig Smith on X: https://x.com/craigss
Eye on A.I. on X: https://x.com/EyeOn_AI

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(386)

Chat GPT's Creator Explains The Shocking Truth About How AI Understands the World | Ilya Sutskever

Chat GPT's Creator Explains The Shocking Truth About How AI Understands the World | Ilya Sutskever

Craig Smith sits down with Ilya Sutskever - Co-Founder and Chief Scientist at Safe Superintelligence Inc. & former chief scientist at OpenAI and one of the primary minds behind GPT-3, GPT-4, and the d...

9 Loka 41min

The Coordination Tax: Why AI Is Burning Out Your Best People | Dan O'Connell, Front

The Coordination Tax: Why AI Is Burning Out Your Best People | Dan O'Connell, Front

Everyone assumes AI in customer operations means fewer people and lower costs. The data from 700 customer operations leaders is more complicated, and more honest. Dan O'Connell, CEO of Front, joins Cr...

6 Loka 51min

Ukraine's Secret Weapon: The Points System Winning the Drone War | Andrii Hrytseniuk

Ukraine's Secret Weapon: The Points System Winning the Drone War | Andrii Hrytseniuk

Ukraine tracks every confirmed drone kill with video evidence, deduplicates the data to prevent double-counting, converts verified strikes into e-points, and delivers newly ordered weapons to frontlin...

2 Loka 33min

Why Current AI Cannot Be Conscious | Dr. Christof Koch

Why Current AI Cannot Be Conscious | Dr. Christof Koch

One in four patients currently being considered for life support withdrawal may actually be fully conscious, they just can't signal it. That single finding, from a landmark New England Journal of Medi...

30 Syys 1h 1min

The Technology for Fully Autonomous Attack Is Already Here | Alex Liannyi, NORDA Dynamics

The Technology for Fully Autonomous Attack Is Already Here | Alex Liannyi, NORDA Dynamics

The AI systems guiding Ukrainian combat drones aren't running on expensive Nvidia chips. They're running on a Raspberry Pi Zero - a $15 hobby computer - and that single detail tells you more about how...

24 Syys 19min

Inside Ukraine's Drone War: Maj. "Phoenix" of Lasar's Group

Inside Ukraine's Drone War: Maj. "Phoenix" of Lasar's Group

The first armed drone Ukraine ever fielded wasn't built in a factory or procured from a defense contractor. It was built in four months by a network engineer using a Starlink terminal and a large agri...

21 Syys 1h 5min

The Hidden Algorithm That Decides Which Software AI Will Recommend | Tim Sanders, G2

The Hidden Algorithm That Decides Which Software AI Will Recommend | Tim Sanders, G2

Most companies investing in AI visibility are optimizing for the wrong thing. Being cited by an AI response and being recommended by an AI response are completely different outcomes, with click-throug...

14 Syys 58min

The Reason 30 Years of Cybersecurity Has Failed - and What Actually Fixes It | Trent Telford, Qanapi

The Reason 30 Years of Cybersecurity Has Failed - and What Actually Fixes It | Trent Telford, Qanapi

Every major data breach in the last 30 years shares the same root cause: the data inside the wall was never protected, only the wall. And AI frontier models are now making that wall easier to breach t...

10 Syys 55min