Good Search Borrows, Great Search … Steals?

Good Search Borrows, Great Search … Steals?

Web crawling—the act of indexing information across the internet—has been around for decades. It has primarily been used by search engines like Google and nonprofits like Internet Archive and Common Crawl to catalog the contents of the open internet and make it searchable. Until recently, the practice of web crawling has rarely been seen as controversial, as websites depended on the process as a way for people to find their content. But now crawling tech has been subsumed by the great AI-ening of everything, and is being used by companies like Google and Perplexity AI to absorb whole articles that are fed into their summarizing machines.

This week on Gadget Lab, WIRED senior writer Kate Knibbs joins the show to talk about web crawling and the controversy over Common Crawl. Then we talk with Forbes’ chief content officer and editor Randall Lane about how Perplexity.AI repurposed a Forbes article and presented it as its own story, without first asking permission or properly citing the source.

Show Notes:

Read Kate’s story about how publishers are going after Common Crawl over AI training data. Read Randall’s story about how Preplexity.AI copied the work of two Forbes reporters.

Recommendations:

Randall recommends his new horse racing league, the National Thoroughbred League. Kate recommends the book Victim by Andrew Boryga. Lauren recommends the show Hacks on Max.

Randall Lane can be found on social media @RandallLane. Kate Knibbs is @Knibbs. Lauren Goode is @LaurenGoode. Michael Calore is @snackfight. Bling the main hotline at @GadgetLab. The show is produced by Boone Ashworth (@booneashworth). Our theme music is by Solar Keys.

Learn about your ad choices: dovetail.prx.org/ad-choices

Jaksot(428)

Iran Strikes in the AI Era; Prediction Markets Ethics; Paramount Beats Netflix

Iran Strikes in the AI Era; Prediction Markets Ethics; Paramount Beats Netflix

This week, the team dives into why disinformation and the AI industry battles have quickly positioned themselves at the center of the ongoing conflict between the U.S and Iran. They also discuss how p...

5 Maalis 33min

BIG INTV: Open AI’s Former Safety Lead Calls Out Erotica Claims (Rerun)

BIG INTV: Open AI’s Former Safety Lead Calls Out Erotica Claims (Rerun)

Steven Adler used to lead product safety at OpenAI. When Katie read his recent op-ed asking OpenAI to prove that they have and continue to address safety issues, she knew she wanted to talk to him. Th...

3 Maalis 39min

Pentagon v. ‘Woke’ Anthropic; Agentic v. Mimetic; Trump v. the State of the Union

Pentagon v. ‘Woke’ Anthropic; Agentic v. Mimetic; Trump v. the State of the Union

This week, the team dives into the feud that has been brewing between Anthropic and the Pentagon — and what it says about how the government interacts with tech companies. Later, Zoë tells us why figu...

26 Helmi 31min

BIG INTV: Melinda French Gates on Secrets: ‘Live a Truthful Life, Then You Don’t Have Any’ (Re-Run)

BIG INTV: Melinda French Gates on Secrets: ‘Live a Truthful Life, Then You Don’t Have Any’ (Re-Run)

On this week’s episode of The Big Interview podcast, the philanthropist sits down with Katie to offer her insights on billionaire donors, kids on phones, and the importance of women’s healthcare.Join ...

24 Helmi 47min

AI Researcher Resignations; Bots Hiring Humans; Evie Magazine’s Party

AI Researcher Resignations; Bots Hiring Humans; Evie Magazine’s Party

This week, Zoë dives into why some researchers at top AI companies have been resigning and publicly voicing their concerns around AI safety. Then, Brian tells us about Rent-A-Human — a website where A...

19 Helmi 30min

ICE’s Secret Expansion Plans; Palantir Employees’ Ethical Concerns; AI Assistants

ICE’s Secret Expansion Plans; Palantir Employees’ Ethical Concerns; AI Assistants

This week, Brian, Leah, and Zoë discuss WIRED’s big scoop on ICE’s startling plans to expand to nearly every state in the US. Then, they unpack Alex Karp’s nearly-hour-long non-response to Palantir em...

12 Helmi 37min

Tech Elites in the Epstein Files; Musk’s Mega Merger; Crypto Scam Compound

Tech Elites in the Epstein Files; Musk’s Mega Merger; Crypto Scam Compound

This week, Brian and Leah dive into the key tech industry figures who show up in the final batch of the Epstein Files. Then, they discuss SpaceX and xAI’s blockbuster merger, and what it says about th...

5 Helmi 35min

BIG INTV: Fallout Director Jonathan Nolan Says He's Still a Techno-Optimist

BIG INTV: Fallout Director Jonathan Nolan Says He's Still a Techno-Optimist

Katie sits down with the Westworld showrunner, Jonah Nolan for a wide ranging conversation about what keeps him coming back to sci-fi storytelling about technology and human nature. After decades of w...

3 Helmi 48min