This Week in AI Security - 17th September 2026

A shorter episode this week, recorded from the sidelines of Mind The Sec, soon to be Black Hat Latin America. The headline is distillation. Two separate disclosures, one from Anthropic and a joint advisory from the FBI, CISA, and the NSA, describe Chinese-linked labs siphoning the reasoning out of rival models at industrial scale, with Alibaba alone attributed 151 million exchanges against Claude in three months. Around that sit the usual escalations: AI-run exploit foundries, malware that rebuilds itself to dodge antivirus, model keys stolen through gateways, a covert attack that hides in plain English, and one runaway agent that burned $50,000 in a couple of hours.

Note: there is no episode next week. We will be back in a couple of weeks with a longer edition covering both weeks.

Key Discussion Points

• Two disclosures detail industrial-scale distillation campaigns from Chinese-linked labs. A joint FBI/CISA/NSA advisory names six companies going back to 2024, and Anthropic reports its largest-ever measured attack: 151 million exchanges attributed to Alibaba alone, peaking at 3 million a day across 3,500 accounts. No exploits, just reframing tactics that coax reasoning traces out of the model.
• Anthropic's latest misuse report covers an autonomous exploit foundry that produced 12 potential zero-days in a month across 50 targets, plus Russian-aligned actors whose agents rebuilt malware to evade antivirus and dumped 2,100 cloud access tokens across 40 tenants in 34 hours.
• Eval sandboxes and LiteLLM gateways are the new target for model-key theft. One actor planted instructions in a vendor's eval sandbox to force key handover, then hit 30 AI companies in four days. Separately, 10% of internet-exposed LiteLLM instances still accept the default admin key.
• A new CVE in IBM's Context Forge MCP gateway allows OS command injection, with a second CVE around default credentials. MCP gateways are becoming a choke point.
• Check Point's "Puzzle Mask" shows that plain, well-crafted English can smuggle a hidden instruction past fast gatekeeper models. It hit a 100% bypass rate against four gatekeepers with 23 prompts, exploiting the fast-gatekeeper, strong-target design many production systems rely on.
• Anthropic CEO Dario Amodei published a "We Must Pace the Frontier" paper calling to slow AI capability improvement, against a backdrop of no real regulation and a narrow antitrust waiver that lets labs coordinate on safety.
• A late-breaking story: one runaway agent running a business-automation process racked up a $50,000 bill in a couple of hours. Most organizations have neither cost guardrails nor full telemetry across hybrid cloud to catch it.

Episode Links

https://techcrunch.com/2026/09/10/anthropic-details-distillation-campaigns-from-alibaba-moonshot-ai-and-deepseek/

https://media.defense.gov/2026/Sep/08/2003992823/-1/-1/1/CSA_CHINA_BASED_AI_COMPANIES_MALICIOUS_DISTILLATION_AGAINST_US.PDF

https://www.anthropic.com/threat-intelligence-report-september-2026

https://www.beri.net/article/anthropic-threat-report-eval-sandbox-litellm-prompt-injection-api-key-theft

https://vulners.com/pypa/PYPA:PYSEC-2026-3862

https://research.checkpoint.com/2026/puzzlemask-abusing-plain-prose-as-a-covert-ai-attack-vector/

https://techcrunch.com/2026/09/12/anthropic-ceo-outlines-plan-to-pace-the-frontier/