Testing AI: Engineering Confidence in Non-Deterministic Systems with Jason Arbon

In this episode 600 of the TestGuild Automation Podcast, Joe Colantonio talks with Jason Arbon, founder of Testers.ai, Jank.AI and IcebergQA and author of the new book Testing AI: Engineering Confidence in Non-Deterministic Systems.

Take Our 2027 Survey Now: https://testgld.link/27data

Jason makes a case most testers have not heard yet. Coding is being absorbed by AI. Specification work is thinning out. Product, development, and test roles are converging into one. And when the music stops, the only seat left belongs to the person who can look at what the machine produced and make an evidence backed call on whether it ships. He calls that confidence engineering, and he argues it is not a rebrand of QA. It is what QA was always supposed to be.

Along the way, Joe and Jason get into the containment problem and why alignment, not lockdown, is now the real safety goal. They dig into why testing cost scales quadratically, meaning ten times more generated code creates roughly a hundred times more testing demand. Jason also pushes back hard on skeptics of agentic testing, pointing out that almost nobody has run the obvious experiment of testing a site themselves for a week and comparing their results against what AI finds.

You will also hear Jason's most practical piece of advice in the whole conversation. If you are not running the same suite five times against the same build and looking at the actual results, not just flake, you are not testing seriously in an AI world.

Plus a detour into grokking, the Chinese Room, and Geoffrey Hinton, because it would not be a Jason Arbon episode without one.

Listen up!

Det här avsnittet är hämtat från ett öppet RSS-flöde och publiceras inte av Podme. Det kan innehålla reklam.

Avsnitt(612)

Locust and Azure Load Testing: Create and Debug Load Tests with AI with Lars Holmberg and Abhinav Premsekhar

Performance testing has never been easier to start, or more important to get right. As teams ship more AI generated code, performance problems are slipping through, and they tend to surface during you...

6 Okt 36min

AI Agents Passed the Tests. Real Devices Didn't. with Nick Metcalfe

Cloud device farms make it easy to get started with real device testing. But what happens when you need to scale? Costs climb fast, you wait in a queue for the devices you need, and you lose visibilit...

29 Sep 40min

AI Testing Reality Check: What 13 Founders Refuse to Automate with Joe Colantonio

I asked thirteen people inside AI testing one question: can AI test your software for you? Founders, CTOs, project leads, and quality leaders, many of them selling AI testing tools right now. Not one ...

22 Sep 29min

How to Stay Irreplaceable When Agents Write Your Tests with Jonathon Wright

Is AI going to replace software testers, or make them more valuable than they have ever been? Jonathon Wright says both, and in this episode he does not back down from either position. Jonathon Wright...

15 Sep 34min

AI Performance Testing: How to Scale Agentic AI with Kandasamy Selvaraj

Everything you knew about performance testing changes when the system you're testing is non deterministic. In this episode of the TestGuild Automation Podcast, Joe Colantonio sits down with Kandasamy ...

8 Sep 43min

AI Testing Is Bigger Than You Think, 5 Areas Testers Must Own with Swati Seela

Most testers are stuck arguing about whether AI is coming for their jobs. Swati Seela thinks that's a distraction from a much more useful question: when we say "AI testing," what do we actually mean? ...

1 Sep 38min

Why IoT Testing Proves Manual Testing Never Died with Oleksii Cherkashyn

Manual testing never died. IoT is the proof. In this episode, Joe Colantonio talks with Oleksii Cherkashyn, a QA team lead, automation engineer, and IoT testing specialist who built a complete test ar...

18 Aug 35min