Generative Benchmarking with Kelly Hong - #728

Generative Benchmarking with Kelly Hong - #728

In this episode, Kelly Hong, a researcher at Chroma, joins us to discuss "Generative Benchmarking," a novel approach to evaluating retrieval systems, like RAG applications, using synthetic data. Kelly explains how traditional benchmarks like MTEB fail to represent real-world query patterns and how embedding models that perform well on public benchmarks often underperform in production. The conversation explores the two-step process of Generative Benchmarking: filtering documents to focus on relevant content and generating queries that mimic actual user behavior. Kelly shares insights from applying this approach to Weights & Biases' technical support bot, revealing how domain-specific evaluation provides more accurate assessments of embedding model performance. We also discuss the importance of aligning LLM judges with human preferences, the impact of chunking strategies on retrieval effectiveness, and how production queries differ from benchmark queries in ambiguity and style. Throughout the episode, Kelly emphasizes the need for systematic evaluation approaches that go beyond "vibe checks" to help developers build more effective RAG applications. The complete show notes for this episode can be found at https://twimlai.com/go/728.

Jaksot(780)

Building an Immune System for AI Generated Software with Animesh Koratana - #746

Building an Immune System for AI Generated Software with Animesh Koratana - #746

Today, we're joined by Animesh Koratana, founder and CEO of PlayerZero to discuss his team’s approach to making agentic and AI-assisted coding tools production-ready at scale. Animesh explains how rap...

9 Syys 20251h 5min

Autoformalization and Verifiable Superintelligence with Christian Szegedy - #745

Autoformalization and Verifiable Superintelligence with Christian Szegedy - #745

In this episode, Christian Szegedy, Chief Scientist at Morph Labs, joins us to discuss how the application of formal mathematics and reasoning enables the creation of more robust and safer AI systems....

2 Syys 20251h 11min

Multimodal AI Models on Apple Silicon with MLX with Prince Canuma - #744

Multimodal AI Models on Apple Silicon with MLX with Prince Canuma - #744

Today, we're joined by Prince Canuma, an ML engineer and open-source developer focused on optimizing AI inference on Apple Silicon devices. Prince shares his journey to becoming one of the most prolif...

26 Elo 20251h 10min

Genie 3: A New Frontier for World Models with Jack Parker-Holder and Shlomi Fruchter - #743

Genie 3: A New Frontier for World Models with Jack Parker-Holder and Shlomi Fruchter - #743

Today, we're joined by Jack Parker-Holder and Shlomi Fruchter, researchers at Google DeepMind, to discuss the recent release of Genie 3, a model capable of generating “playable” virtual worlds. We dig...

19 Elo 20251h 1min

Closing the Loop Between AI Training and Inference with Lin Qiao - #742

Closing the Loop Between AI Training and Inference with Lin Qiao - #742

In this episode, we're joined by Lin Qiao, CEO and co-founder of Fireworks AI. Drawing on key lessons from her time building PyTorch, Lin shares her perspective on the modern generative AI development...

12 Elo 20251h 1min

Context Engineering for Productive AI Agents with Filip Kozera - #741

Context Engineering for Productive AI Agents with Filip Kozera - #741

In this episode, Filip Kozera, founder and CEO of Wordware, explains his approach to building agentic workflows where natural language serves as the new programming interface. Filip breaks down the ar...

29 Heinä 202546min

Infrastructure Scaling and Compound AI Systems with Jared Quincy Davis - #740

Infrastructure Scaling and Compound AI Systems with Jared Quincy Davis - #740

In this episode, Jared Quincy Davis, founder and CEO at Foundry, introduces the concept of "compound AI systems," which allows users to create powerful, efficient applications by composing multiple, o...

22 Heinä 20251h 13min

Building Voice AI Agents That Don’t Suck with Kwindla Kramer - #739

Building Voice AI Agents That Don’t Suck with Kwindla Kramer - #739

In this episode, Kwindla Kramer, co-founder and CEO of Daily and creator of the open source Pipecat framework, joins us to discuss the architecture and challenges of building real-time, production-rea...

15 Heinä 20251h 13min

Suosittua kategoriassa Politiikka ja uutiset

uutiscast
aikalisa
rss-ootsa-kuullut-tasta
politiikan-puskaradio
ootsa-kuullut-tasta-2
tervo-halme
viisupodi
rss-podme-livebox
otetaan-yhdet
et-sa-noin-voi-sanoo-esittaa
rikosmyytit
the-ulkopolitist
rss-asiastudio
io-techin-tekniikkapodcast
aihe
rss-pallo-keskelle-2
radio-antro
rss-kovin-paikka
rss-sanna-ukkola-show-verkkouutiset
rss-tasta-on-kyse-ivan-puopolo-verkkouutiset