Databricks broadcast joins: when the memo beats the meeting (and when it kills a task)

Two engineers on the same team join the same big orders table to the same small lookup table. Same cluster, same data, one line of code different. Sarah's finishes in the time it takes to get a coffee. Mike's dies, and the error isn't about the data at all. A task ran out of memory building a hash map.

Broadcast joins get passed around as a tip instead of a model: small table equals fast, flip the switch when a join runs long. But the mechanism in almost every write-up is years out of date, and the size Spark checks is not the size that has to fit in memory.

In this episode:

- How to explain broadcast versus shuffle joins at standup, in one sentence, without a whiteboard

- What actually happens on a current Databricks runtime when a join broadcasts, and why the popular explanation stopped being true

- Why the size estimate Spark trusts is not the size that lands, and where to read the real payload

- What adaptive query execution rescues you from, and the two places its hands are tied

- The question to ask before you add a broadcast hint, and the failure signature that tells you a broadcast is what broke

This episode is for Databricks data engineers who write joins every week and treat the broadcast hint as a speed switch. Whether you're mid-level and tired of guessing why one join flies and an identical one falls over, or senior and about to be asked in an interview why Spark chose a sort merge join, you'll walk away able to predict the call before you run the query.

---

Helping 18,000+ Databricks data engineers become seniors: interview like seniors, execute like seniors, think like seniors.

Follow The Databricks Data Engineer for new episodes every week.

LinkedIn: linkedin.com/in/jrlasak

Newsletter: dataengineer.wiki

#DataEngineering #Databricks #DataEngineer #CareerGrowth #ApacheSpark #DeltaLake1