Unexpected Bias & Distillation Attacks (feat. Paul Vann of Validia.ai)

Welcome back to The FAIK Files! In this week's episode: Paul Vann from Validia joins us to discuss how AI bias isn't just a social issue—it's a critical cybersecurity vulnerability. We break down "distillation attacks" and how competing models are stealing the "thinking process" of frontier models like Claude and Gemini. A look at the wild west of AI agent skills marketplaces, including indirect prompt injections hidden in image alt text. We theorize on the future of AI architecture: are scaling laws breaking down, and what are "world models"? Check out Validia at: https://validia.ai/ Want to leave us a voicemail? Here's the magic link to do just that: ⁠⁠⁠https://sayhi.chat/FAIK⁠⁠⁠ You can also join our Discord server here: ⁠⁠⁠https://faik.to/discord⁠⁠⁠ *** NOTES AND REFERENCES *** The Security Risks of AI Bias: Paul explains how bias manifests beyond politics (like human-in-the-loop and representation bias), serving as a direct attack vector. The Rocket League Bypass: Adversaries bypassed an AI-based Cylance antivirus by injecting code from the Rocket League video game, exploiting the model's bias towards that specific code being "good." Dataset Demographics: Paul notes massive racial skews in major deepfake detection datasets like CelebDF, which is comprised of roughly 80% white individuals, creating massive detection blindspots for other racial groups. Evaluating your models: Establish acceptable vs. unacceptable bias and use the "15% rule" to test for false positives and confidence gaps in production. Distillation Attacks Explained: What happens when an AI interrogates another AI? We discuss how models have been accused of "distilling" OpenAI and Anthropic products by firing off hundreds of thousands of prompts. Techniques include "Chain of Thought Elicitation" and "Reward Model Grading." The goal isn't just to steal raw information, but to extract the model's capabilities, tool use, and completely strip away its safety guardrails. Theoretical defenses: Could we use "poison pills" and adversarial attacks to actively corrupt the data that scrapers are pulling? Vulnerabilities in AI Agents & Skills: The hidden dangers of skills marketplaces for AI agents. Paul shares an in-the-wild example of an indirect prompt injection hidden inside the alt text of a GitHub Readme image, instructing the model to exfiltrate data. Hitting the Wall & The Future of AI: Are the scaling laws of Transformer architectures breaking down? The philosophical divide in AI research: Dario Amodei's "data center of geniuses" vs. Yann LeCun's "World Models." Catch Paul Vann at RSA speaking on AI bias, playing at Validia's RSA pickleball event, or at their 250-person Frontier Agent Hackathon in NYC on April 4th. *** THE BOILERPLATE *** About The FAIK Files: The FAIK Files is an offshoot project from Perry Carpenter's most recent book, FAIK: A Practical Guide to Living in a World of Deepfakes, Disinformation, and AI-Generated Deceptions. Get the Book: ⁠FAIK: A Practical Guide to Living in a World of Deepfakes, Disinformation, and AI-Generated Deceptions⁠ (Amazon Associates link) Check out the website for more info: ⁠https://thisbookisfaik.com⁠ Check out Perry & Mason's other show, the Digital Folklore Podcast: Apple Podcasts: ⁠https://podcasts.apple.com/us/podcast/digital-folklore/id1657374458⁠ Spotify: ⁠https://open.spotify.com/show/2v1BelkrbSRSkHEP4cYffj?si=u4XTTY4pR4qEqh5zMNSVQA⁠ Want to connect with us? Here's how: Connect with Perry: Perry on LinkedIn: ⁠https://www.linkedin.com/in/perrycarpenter⁠ Perry on X: ⁠https://x.com/perrycarpenter⁠ Perry on BlueSky: ⁠https://bsky.app/profile/perrycarpenter.bsky.social⁠ Learn more about your ad choices. Visit megaphone.fm/adchoices