Syntax - Tasty Web Development Treats1 Mai 2024

763: Web Scraping + Reverse Engineering APIs

Web scraping 101! Dive into the world of web scraping with Scott and Wes as they explore everything from tooling setup and navigating protected routes to effective data management. In this Tasty Treat episode, you’ll gain invaluable insights and techniques to scrape (almost) any website with ease. Show Notes 00:00 Welcome to Syntax! 03:13 Brought to you by Sentry.io. 05:00 What is scraping? Socialblade. 08:01 Examples of past scrapers. Canadian Tire. 10:06 Cloud app downloader. 16:13 Other use cases. 16:58 Scraping 101. 17:28 Client Side. 19:08 Private API. Proxyman. 22:40 Server rendered. 23:27 Initial state. 24:57 What format is the data in? Google Puppeteer Extension. 27:08 Working with the DOM. 27:12 Linkedom npm package. 29:02 querySelector everything. 31:28 How to find the elements without classes. 34:08 Use XPath selectors for select by word. 34:53 Make them as flexible as you can. Classes change! 35:10 AI is good at this! 36:26 File downloading. 38:20 Working with protected routes. Proxyman. 40:41 Programatically retrieve authentication keys because they are short-lived. Fetch Cookie. 43:20 Deal-breakers. Mechanical Turk. 44:58 What happened with Amazon? Uniqlo Self-Checkout 46:42 Wes’ portable refrigerator utopia. 47:25 Sick Picks & Shameless Plugs. Sick Picks Scott: KeyboardCleanTool. Wes: Yabai. Shameless Plugs Scott: Syntax on YouTube Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads Wes: X Instagram Tiktok LinkedIn Threads Scott:X Instagram Tiktok LinkedIn Threads Randy: X Instagram YouTube Threads

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(1050)

1050 - Everyone Is Hacking Their TV

Wes, Scott, and CJ run through the week’s biggest web dev news: everyone’s hacking their TVs, VoidZero open-sourced its Void framework with Cloudflare deploys, OpenCode got hacked, and Anthropic built...

26 Okt 1h 19min

1049: WTF is a Software Factory?

Scott and Wes talk with Adam Jacob about software factories: what they actually are, why they’re unique to each team, and how to build trust in agent-generated code. They dig into Swamp Club, black bo...

21 Okt 57min

1048: Deno Is Dead

Deno is shutting down and the team is heading to Cloudflare — Scott, Wes, and CJ break down what that means for the runtime, JSR, and workerd. Plus OpenCode 2.0 on durable objects, TanStack Start 1.0,...

19 Okt 1h 40min

1047: Adobe has a big problem

Someone just clean-room ported seven Adobe apps to Rust, and Wes is shocked at how good it is. Scott and Wes break down what it gets right, what’s still missing, and why copying software is easy but i...

14 Okt 35min

1046: The Pi Updates are insane

Scott, Wes, and CJ break down the latest Pi updates, including durable long-running agents, Codemode, and MCP support, then debate whether pull requests are dead in favor of issues. Plus Supabase pick...

12 Okt 1h 40min

1045: Who can actually max out 5 AI subscriptions?

Scott and Wes answer listener questions about what replaced blogging, what to do while your AI agent works, and who should pay for token costs. They also cover keeping agents from going off the rails,...

7 Okt 52min

1044: Ruby on Rails is Dead

Scott, Wes, and CJ ask whether Ruby on Rails is really dead (and whether Rust is just Rails you don't have to read), then dig into Meta's new Muse agent, which is free, comes with its own VM, and is a...

5 Okt 1h 20min

1043: I’m Using GPUI For Everything

Scott and Wes ditch Electron and Tauri for GPUI, the GPU accelerated Rust UI framework from the Zed team. They get into its React-like API, styling that feels a lot like Tailwind, GPUI Kit components,...

30 Sep 32min