AI Agents Passed the Tests. Real Devices Didn't. with Nick Metcalfe

Cloud device farms make it easy to get started with real device testing. But what happens when you need to scale? Costs climb fast, you wait in a queue for the devices you need, and you lose visibility into what's actually happening when a test fails.

In this episode, Nick Metcalfe, Head of Technical Sales Engineering at Cambrionix, joins Joe to explain why more testing teams are bringing their device labs in house. Nick shares how one team was quoted roughly a million dollars a month to outsource their real device testing, then cut that cost to around $300K by building their own lab. And cost is only part of the story.

You'll discover why an in house device lab can outperform a cloud lab:

• Lower cost at scale. Cloud pricing works for small test runs, but at enterprise scale an in house lab can save hundreds of thousands of dollars every month.
• No waiting in line. Your devices are yours, available whenever you need them, with no queue and faster turnaround.
• Full control and visibility. See exactly what's happening at the hardware level, down to the millisecond a USB controller reboots, so you can tell a real bug from an infrastructure failure.
• Test the devices your users actually have. Keep legacy phones, budget devices, and hand me downs in your lab, just like Snap does to support its user base across 12,000+ devices and 700,000 tests a day.
• Test custom hardware. Point of sale systems, VR headsets, smart devices, and prototypes that a cloud vendor simply won't have.
• Automate the hands on work. Reboot devices, recover from restore mode, and put 16 Apple devices into DFU mode at once, all through an API that plugs into your CI pipeline.
• Protect your device investment. Hold batteries at a set charge level to reduce wear and extend device life for years.

Nick also covers the honest tradeoffs, including the setup and resources an in house lab requires, and why a hybrid approach of emulators first and real devices second gives you the most confidence before release. Plus, his number one tip for building a lab: do it once, do it right.

If your team is scaling mobile test automation, fighting flaky tests, or questioning your cloud device farm bill, this episode is for you.