Why Stack Overflow and Cloudflare launched a pay-per-crawl model

In this episode of Leaders of Code, Stack Overflow’s Janice Manningham and Josh Zhang sit down with Cloudflare VP Will Allen to discuss the innovative pay-per-crawl model co-launched by their organizations. They explore how the rise of AI has disrupted the traditional “open versus block” internet model, creating a need for platforms to protect their content and data from commercial exploitation while maintaining community access.

The discussion also:

Explores the future of the bot ecosystem, emphasizing the importance of putting publishers back in the driver’s seat to decide how their content is accessed and monetized.
Explains the technical implementation of the pay-per-crawl system, which uses Cloudflare’s bot categorization and WAF rules to serve a 402 “Payment Required” message to specific crawlers.
Highlights the strategic value of data licensing, comparing comprehensive enterprise contracts with the more flexible, programmatic pay-per-use access enabled by the new model.

Notes

Connect with Will Allen, Janice Manningham and Josh Zhang on LinkedIn.
Learn more about Stack Overflow Data Licensing here.

See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(975)

From PHP to team lead of agents: rethinking judgment, review, and data with Google's Andi Gutmans (Part 1)

Andi Gutmans, head of Agentic Data Cloud at Google and co-creator of PHP, joins Leaders of Code to talk about why agentic development feels less like a break from the past and more like the next chapt...

21 Aug 25min

Get rid of your CAPTCHA, the future of the web is bots

21 Aug 32min

Building an agentic SDLC with a QA engineering mindset

Ryan welcomes Suneet Malhotra, Senior Manager of Test Engineering at Motorola Solutions, to chat about building end-to-end agentic SDLC pipelines using MCPs, using Cohen’s kappa to evaluate multiple L...

18 Aug 28min

Solving integration woes with a hackathon

Ryan welcomes Meryll Blanchet, Director of Engineering for Adobe Brand Visibility, to chat about Adobe’s recent acquisition of Semrush, how Adobe Brand Visibility was born from Semrush’s AI visibilit...

14 Aug 28min

Your tokenmaxxing is not valuemaxxing

Ryan is joined by Coder’s Rob Whiteley to chat about why tokenmaxxing isn’t proving real value and just triggering Goodhart’s Law, how release speed and PR merges can help you measure agentic outcomes...

11 Aug 29min

How to be fearlessly AI native

Ryan welcomes McLaren Stanley, Senior Principal Engineer for Amazon Stores, to discuss what it actually takes to make teams AI native, why agentic engineering is shifting code bottlenecks downstream t...

7 Aug 40min

Your MVP doesn’t need a Kubernetes cluster

Ryan welcomes Anurag Goel, CEO and co-founder of Render, to discuss why most startups shouldn’t start by managing their Kubernetes and cloud infrastructure. They discuss how frustration with spinning ...

4 Aug 31min

What happens to the internet when robots act like humans?

Ryan welcomes WPEngine CTO Ramadass Prabakar to the show to chat about what happens—and what we should do—when agents start acting like humans online, how our internet is evolving to serve both human ...

31 Jul 30min