AI Testing Reality Check: What 13 Founders Refuse to Automate with Joe Colantonio

I asked thirteen people inside AI testing one question: can AI test your software for you? Founders, CTOs, project leads, and quality leaders, many of them selling AI testing tools right now. Not one said yes.

This episode is my session from the Testear.la 2026 conference, built entirely from twelve months of interviews on this show with the people actually building this technology, including Applitools, Maestro, Bright Security, QA.tech, Cucumber, Ito, and the lead of the Selenium project.

In it you'll learn why one AI testing vendor deliberately removed AI from the part where tests run, the real productivity number teams are measuring versus the number in the case studies, the four incompatible architectural bets vendors are making and how to tell which one your tool has chosen for you, why AI generated code is producing vulnerabilities far faster than it produces code, and the one architecture rule that stops an agent from editing the test that was supposed to catch it.

You'll also get the list. Every task these thirteen people said they would never delegate, assembled from conversations that had no coordination between them. Plus two tests you can run on Monday, both free, that tell you whether you're ready to trust an agent with your suite.

If you're a tester, automation engineer, or QA leader trying to figure out where you fit in all of this, this one is for you. The people building AI testing don't trust it to test on its own. That gap is your job.

Exclusive Sponsor

Quick note before we get into this one.

Everything you're about to hear points at the same conclusion. The work is moving from writing tests to supervising the things that write them. That's a different skill set, and most of us are picking it up by accident.

Automation Guild '27 is where you pick it up on purpose. Year eleven. February 8th through the 12th. Five days, live, with the best E2E automation practitioners working today, and you keep lifetime access to every session.

The early bird list is open right now, and that's where the biggest discount goes.

Head to testguild.com and look for Automation Guild '27 early bird, or grab the link in the show notes: https://testgld.link/yearlybird