AI + a16z28 Maalis 2025

Benchmarking AI Agents on Full-Stack Coding

In this episode, a16z General Partner Martin Casado sits down with Sujay Jayakar, co-founder and Chief Scientist at Convex, to talk about his team’s latest work benchmarking AI agents on full-stack coding tasks. From designing Fullstack Bench to the quirks of agent behavior, the two dig into what’s actually hard about autonomous software development, and why robust evals—and guardrails like type safety—matter more than ever. They also get tactical: which models perform best for real-world app building? How should developers think about trajectory management and variance across runs? And what changes when you treat your toolchain like part of the prompt? Whether you're a hobbyist developer or building the next generation of AI-powered devtools, Sujay’s systems-level insights are not to be missed.

Drawing from Sujay’s work developing the Fullstack-Bench, they cover:

Why full-stack coding is still a frontier task for autonomous agents
How type safety and other “guardrails” can significantly reduce variance and failure
What makes a good eval—and why evals might matter more than clever prompts
How different models perform on real-world app-building tasks (and what to watch out for)
Why your toolchain might be the most underrated part of the prompt
And what all of this means for devs—from hobbyists to infra teams building with AI in the loop

Learn More:

Introducing Fullstack-Bench

Follow everyone on X:

Sujay Jayakar

Martin Casado

Check out everything a16z is doing with artificial intelligence here, including articles, projects, and more podcasts.

Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(100)

Ideogram’s Open-Weights Image Model and the Future of AI Design

Yoko Li and Justine Moore speak with Ideogram founder and CEO Mohammad Norouzi about image generation models, design workflows, and the evolving relationship between AI and creative work. The conversa...

15 Kesä 42min

Building Search for AI Agents with Exa CEO Will Bryk

Sarah Wang speaks with Exa cofounder and CEO Will Bryk about building search infrastructure for the AI era. The conversation covers Exa’s origins, why traditional search engines were not designed for ...

4 Kesä 49min

AI Agents and the Fight for Customer Data

Martin Casado speaks with George Fraser, cofounder and CEO of Fivetran, about the future of data infrastructure in the age of AI. The conversation covers Fivetran’s merger with dbt, the changing role ...

2 Kesä 50min

Ben Horowitz on AI Infrastructure, Economics and The New Laws of Software

Recorded live at the a16z Fintech Connect conference in Deer Valley, Alex Rampell speaks with Ben Horowitz, cofounder and general partner at a16z, about how AI has rewritten the fundamental rules of s...

19 Touko 29min

AI Infrastructure, Distribution, and the Next Wave of Software

Sophie Buonassisi speaks with Jennifer Li, general partner at a16z, about why infrastructure is becoming one of the most important areas in AI. They discuss how the shift to AI-native systems is resha...

12 Touko 38min

From Vector Databases to Knowledge Engines: The Next Layer of AI

Peter Levine speaks with Ash Ashutosh, CEO of Pinecone, about the launch of Nexus and the shift from vector databases to knowledge engines. As agents become the primary users of software, they discuss...

5 Touko 46min

Why We Need Continual Learning

Elena Burger speaks with Malika Aubakirova, partner on the AI infrastructure team at a16z, about why today’s AI systems struggle to learn over time. They discuss the limits of in-context learning, the...

28 Huhti 18min

The Agent Era: Building Software Beyond Chat with Box CEO Aaron Levie

Erik Torenberg, Steve Sinofsky, and Martin Casado speak to Aaron Levie, CEO at Box, about what happens to enterprise software when agents become the primary users. They discuss why coding agents succe...

21 Huhti 59min