Generative Benchmarking with Kelly Hong - #728

Generative Benchmarking with Kelly Hong - #728

In this episode, Kelly Hong, a researcher at Chroma, joins us to discuss "Generative Benchmarking," a novel approach to evaluating retrieval systems, like RAG applications, using synthetic data. Kelly explains how traditional benchmarks like MTEB fail to represent real-world query patterns and how embedding models that perform well on public benchmarks often underperform in production. The conversation explores the two-step process of Generative Benchmarking: filtering documents to focus on relevant content and generating queries that mimic actual user behavior. Kelly shares insights from applying this approach to Weights & Biases' technical support bot, revealing how domain-specific evaluation provides more accurate assessments of embedding model performance. We also discuss the importance of aligning LLM judges with human preferences, the impact of chunking strategies on retrieval effectiveness, and how production queries differ from benchmark queries in ambiguity and style. Throughout the episode, Kelly emphasizes the need for systematic evaluation approaches that go beyond "vibe checks" to help developers build more effective RAG applications. The complete show notes for this episode can be found at https://twimlai.com/go/728.

Jaksot(779)

Bridging the Gap Between Academic and Industry Careers with Ross Fadely - TWiML Talk #68

Bridging the Gap Between Academic and Industry Careers with Ross Fadely - TWiML Talk #68

We close out our NYU Future Labs AI Summit interview series with Ross Fadely, a New York based AI lead with Insight Data Science. Insight is an interesting company offering a free seven week post-doct...

16 Marras 201719min

The Limitations of Human-in-the-Loop AI with Dennis Mortensen - TWiML Talk #67

The Limitations of Human-in-the-Loop AI with Dennis Mortensen - TWiML Talk #67

We continue our NYU Future Labs AI Summit interview series with Dennis Mortensen, founder and CEO of X.ai, a company whose AI-based personal assistant Amy helps users with scheduling meetings. I caugh...

13 Marras 201735min

Nexus Lab Cohort 2 - Second Mind - TWiML Talk #66

Nexus Lab Cohort 2 - Second Mind - TWiML Talk #66

The podcast you’re about to hear is the fourth of a series of shows recorded at the NYU Future Labs AI Summit last week in New York City. In this show, I speak with Kul Singh, CEO and Founder of Secon...

9 Marras 201721min

Nexus Lab Cohort 2 - Bite.ai - TWiML Talk #65

Nexus Lab Cohort 2 - Bite.ai - TWiML Talk #65

The podcast you’re about to hear is the second of a series of shows recorded at the NYU Future Labs AI Summit last week in New York City.In this episode, you’ll hear from Bite.ai, a startup founded by...

8 Marras 201726min

Nexus Lab Cohort 2 - Bowtie - TWiML Talk #64

Nexus Lab Cohort 2 - Bowtie - TWiML Talk #64

The podcast you’re about to hear is the second of a series of shows recorded at the NYU Future Labs AI Summit last week in New York City. In this episode, I speak with Ron Fisher and Mike Wang, who, a...

7 Marras 201725min

AI Nexus Lab Cohort 2 - Mt. Cleverest - TWiML Talk #63

AI Nexus Lab Cohort 2 - Mt. Cleverest - TWiML Talk #63

The podcast you’re about to hear is the first of a series of shows recorded at the NYU Future Labs AI Summit last week in New York City. My guests this time around are James Villarrubia and Bernie Pra...

6 Marras 201732min

Learning to Learn, and other Opportunities in Machine Learning with Graham Taylor - TWiML Talk #62

Learning to Learn, and other Opportunities in Machine Learning with Graham Taylor - TWiML Talk #62

The podcast you’re about to hear is the third of a series of shows recorded at the Georgian Partners Portfolio Conference last week in Toronto. My guest this time is Graham Taylor, professor of engine...

3 Marras 201737min

Building Conversational Application for Financial Services with Kenneth Conroy - TWiML Talk #61

Building Conversational Application for Financial Services with Kenneth Conroy - TWiML Talk #61

The podcast you’re about to hear is the second of a series of shows recorded at the Georgian Partners Portfolio Conference last week in Toronto. My guest for this interview is Kenneth Conroy, VP of da...

1 Marras 201737min

Suosittua kategoriassa Politiikka ja uutiset

aikalisa
rss-ootsa-kuullut-tasta
tervo-halme
ootsa-kuullut-tasta-2
politiikan-puskaradio
viisupodi
et-sa-noin-voi-sanoo-esittaa
otetaan-yhdet
rss-podme-livebox
rss-vaalirankkurit-podcast
rss-asiastudio
the-ulkopolitist
rss-kaikki-uusiksi
rss-tekkipodi
io-techin-tekniikkapodcast
rikosmyytit
rss-mina-ukkola
rss-fingo-podcast
rss-hyvaa-huomenta-bryssel
rss-merja-mahkan-rahat