Episode Five: When AI Decides Who Dies, Who’s Responsible? Autonomous AI Weapons and Red Lines with Matthew Rosenquist

When an autonomous drone picks its own target and kills civilians, who’s responsible? The commander, the developer, or the state? Cybersecurity veteran Matthew Rosenquist (36+ years, advisor to governments and corporations) explains why AI weapons need international red lines now.In this episode of Redefining AI, Lauren Hawker Zafer and Matthew discuss his Help Net Security article on autonomous AI weapons:• How weapons went from AI-assisted to fully autonomous, with no human in the loop after launch• Why a consumer AI vision chip ended up in a Russian drone used against Ukrainian civilians• The “plausible deniability” problem, and how autonomy could make war crimes easier to hide• Why banning autonomous weapons isn’t realistic, but red lines are• How the Geneva Conventions model shows international agreements can work even when not everyone signs

If you've watched generative AI over the past few years, you already understand the shift happening in military hardware. You've just seen it with lower stakes.

Weapons are following the same arc. The first generation of AI in warfare was a very capable assistant. Algorithms crunched telemetry, handled navigation, flagged shapes that looked like tanks but a human operator stayed in the loop, looked at the picture, and made the call. The failure modes were human ones: an overloaded operator, a jammed or intercepted comms link.

You hand it a mission objective and let it go. It plots its own route, coordinates with other autonomous assets, picks targets, and strikes. After launch, there's no one in the loop at all. The failure modes change accordingly: model drift, misidentified targets, and collateral damage delivered at machine speed.

In enterprise software, an agent that goes rogue books the wrong flight or deletes a database. On the battlefield, the error bars are measured in lives.

Here's the part that should make the commercial tech industry uncomfortable: none of this requires exotic, classified hardware anymore.

In his Help Net Security article, Rosenquist points to a case in which an Nvidia vision processor, a chip built for low-power edge computing, self-driving cars, and consumer photography drones, turned up inside a Russian military drone used against Ukrainian civilians in Zaporizhzhia.

His analogy is blunt. "I can design the perfect hammer to build homes for the homeless, but somebody can take that hammer and hit their neighbor in the head and commit murder."

The chip did exactly what it was designed to do: see. The problem is what it was told to look for. And because the same silicon underpins cars, factories, and the drone your cousin flies at weddings, choking it off at the source isn't realistic without kneecapping entire civilian industries. The real vulnerability isn't in the fab. It's in how little software it now takes to turn a benign chip into a weapon that thinks for itself.

The military's "kill chain" is the sequence of decisions that ends with something going boom. For decades, international humanitarian law has rested on a simple assumption about that chain: somewhere in it, a person decided.

Play it forward and you can hear the finger-pointing already. The field commander blames an algorithmic anomaly. The developer says the model performed to spec and the deployment context was unforeseeable. The state shrugs and calls it a tragic edge case. Everyone is technically telling the truth, and nobody is responsible.

Don't forget to share this episode with your friends and subscribe to the show to keep up to date with the latest episodes. And, if you feel our vibe, drop us 5* and help us grow.

With love and light, Redefining AI the Tech Podcast.