Ep.6: How Real-World Messiness Impacts AI Agent Success

Ep.6: How Real-World Messiness Impacts AI Agent Success

In this episode of the Cisco AI Insights Podcast, hosts Rafael Herrera and Sónia Marques are joined by Cisco ML Engineer Paul Mutawe to explore the fascinating paper, "Measuring AI Ability to Complete Long Software Tasks," which introduces a novel time horizon metric to evaluate how autonomously AI agents can execute complex, multi-hour engineering projects. The discussion looks at the rapid evolution of these agents, highlighting the key finding that the fifty percent success time horizon is doubling every two hundred and seven days, while detailing how unbiased benchmarking environments like the Modular Public harness are used to evaluate frontier models alongside real-world complexities like the sixteen-item messiness factor, which significantly reduces agent success rates, and the critical need for human-in-the-loop oversight to combat context rot. A special thank you to the research team at Model Evaluation and Threat Research, who developed this paper. If you are interested in reading the paper yourself, please visit the link: https://arxiv.org/pdf/2503.14499

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(500)

The evolution of good technology | Tech Unscripted

The evolution of good technology | Tech Unscripted

In this episode, industry leaders discuss how to redefine “good” technology today — weighing speed, agility and AI against security, ethics and human impact. Panelists share real examples, from digita...

14 Elo 33min

Scaling AI: The Playbook for IT Services Delivery

Scaling AI: The Playbook for IT Services Delivery

Cisco Technical Assistance Center (TAC) manages 1.4 million SRs annually, and today, 97% are touched by AI. As we shift from task-based automation to Agentic AI, some of these interactions are becomin...

14 Elo 29min

404 Script Not Found: Part 2! AI Deep Dive with Annie Hardy

404 Script Not Found: Part 2! AI Deep Dive with Annie Hardy

Back for part two of our split episode with the founder of Split Futures, Annie Hardy and jump right in where we left off...getting practical about choosing the right AI model—because apparently not e...

13 Elo 24min

404 Script Not Found: AI Deep Dive with Annie Hardy

404 Script Not Found: AI Deep Dive with Annie Hardy

Your company bought all the AI tools and now nobody knows what to do with them. Sound familiar? Kat and Ian sit down with Annie Hardy, founder and Chief AI Architect at Split Futures, to unpack why l...

11 Elo 19min

Cisco Tech Stories Season 4 Kickoff: Top Episodes & Wild Tech Tales

Cisco Tech Stories Season 4 Kickoff: Top Episodes & Wild Tech Tales

Episode 37 kicks off Cisco Tech Stories' fourth season with a highlights reel of the best episodes from the past three years. Hosts Nico and Tristan revisit standout interviews and mini‑series, sharin...

2 Elo 43min

Shaping Cisco’s Portfolio Strategy with Jeetu Patel and Jeff Schultz

Shaping Cisco’s Portfolio Strategy with Jeetu Patel and Jeff Schultz

Jeff Schultz joins Jeetu Patel to delve into the evolution of the Cisco Portfolio and how this focus on critical business outcomes for the AI era is delivering real results for the ecosystem.

30 Heinä 35min

Building Future-Proofed Workplaces with Jeetu Patel and Anurag Dhingra

Building Future-Proofed Workplaces with Jeetu Patel and Anurag Dhingra

Jeetu and Anurag Dhingra explore the transformation of physical infrastructure and intelligent software as the foundation for the workplace. As the nature of work continues to evolve, the demand for e...

30 Heinä 28min

Scaling AI: Enterprise Requirements of Agentic AI

Scaling AI: Enterprise Requirements of Agentic AI

95% of enterprise IT leaders plan to launch agentic AI projects — but 69% aren't ready. So, what's standing in the way?  In this episode of Scaling AI in the Enterprise, we bring together an expert p...

30 Heinä 33min